{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Python对象模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## 类型、对象体系"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T07:30:05.647515Z",
     "start_time": "2021-03-16T07:30:05.636521Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'int'> True\n"
     ]
    }
   ],
   "source": [
    "a = 1\n",
    "print(type(a),isinstance(a, int))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "整数类型的类型还是一种类型，即**类型的类型(元类型)** 。 只是这个类型比较特殊，它的实例对象还是类型对象；\n",
    "Python 中还有一个特殊类型**object**，所有其他类型均继承于object ，换句话讲**object**是所有类型的基类：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_01\\C1_01.png\" width=500 height=400>  \n",
    "\n",
    "完整的继承链如下所示，type是所有类型的元类型，同时type作为对象也继承自object，object是所有类型的基类：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_01\\C1_02.png\" width=500 height=400>  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T07:29:11.325240Z",
     "start_time": "2021-03-16T07:29:11.315246Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'type'> False\n",
      "<class 'type'> True\n",
      "<class 'type'> True True\n",
      "<class '__main__.Dog'>\n",
      "<class 'type'> True True\n",
      "<class '__main__.Sleuth'>\n"
     ]
    }
   ],
   "source": [
    "class Dog(object):\n",
    "\n",
    "    def yelp(self):\n",
    "        print('woof')\n",
    "\n",
    "class Sleuth(Dog):\n",
    "\n",
    "    def hunt(self):\n",
    "        pass\n",
    "\n",
    "dog = Dog()\n",
    "sleuth = Sleuth()\n",
    "\n",
    "print(type(object),issubclass(object,type))\n",
    "print(type(type),issubclass(type,object))\n",
    "print(type(Dog),issubclass(Dog,object),issubclass(Dog,Dog))\n",
    "print(type(dog))\n",
    "print(type(Sleuth),issubclass(Sleuth, object),issubclass(Sleuth,Dog))\n",
    "print(type(sleuth))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## 可变对象 与 不可变对象"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**可变对象**：在对象创建后，其值可以进行修改；\n",
    "\n",
    "**不可变对象**：在对象创建后的整个生命周期，其值都不可修改；"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## 定常对象与不定长对象"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T07:45:09.246384Z",
     "start_time": "2021-03-16T07:45:09.232393Z"
    },
    "hidden": true
   },
   "source": [
    "定常对象如：浮点数，可以用有限的位数在牺牲精度的条件下表示很大的位数；\n",
    "\n",
    "不定长对象，如：整数，字符串对象；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T07:52:58.136257Z",
     "start_time": "2021-03-16T07:52:58.121253Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "28\n",
      "32\n",
      "44\n",
      "\n",
      "*********** Here is string. ************\n",
      "\n",
      "50\n",
      "52\n",
      "75\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "print(sys.getsizeof(1))\n",
    "print(sys.getsizeof(100000000000000000))\n",
    "print(sys.getsizeof(100000000000000000000000000000000000000000000))\n",
    "\n",
    "print('\\n{:*^40}\\n'.format(' Here is string. '))\n",
    "\n",
    "print(sys.getsizeof('a'))\n",
    "print(sys.getsizeof('abc'))\n",
    "print(sys.getsizeof('abcdefghijklmnopqrstuvwxyz'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T07:51:03.492449Z",
     "start_time": "2021-03-16T07:51:03.475459Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24\n",
      "24\n",
      "24\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "print(sys.getsizeof(1.))\n",
    "print(sys.getsizeof(100000000000000000.))\n",
    "print(sys.getsizeof(100000000000000000000000000000000000000000000.))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## 揭开对象神秘的面纱"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "表示对象的结构体有一个**公共头部** \n",
    "在 Python 内部，对象都由 PyObject 结构体表示，对象引用则是指针 PyObject*。 \n",
    "\n",
    "    typedef struct _object {\n",
    "        _PyObject_HEAD_EXTRA\n",
    "        Py_ssize_t ob_refcnt;\n",
    "        struct _typeobject *ob_type;\n",
    "    } PyObject;\n",
    "    \n",
    " **_PyObject_HEAD_EXTRA** :宏\n",
    " \n",
    " **引用计数 ( ob_refcnt )**：Py_ssize_t ob_refcnt;\n",
    " \n",
    " **类型指针 ( ob_type )**：struct _typeobject *ob_type;\n",
    " \n",
    " 引用计数 很好理解：对象被其他地方引用时加一，引用解除时减一； 当引用计数为零，便可将对象回收，这是最简单的垃圾回收机制。\n",
    " \n",
    " 类型指针 指向对象的 类型对象 ，类型对象 描述 实例对象 的数据及行为。\n",
    " \n",
    "  _PyObject_HEAD_EXTRA 宏的定义：\n",
    "\n",
    "    #ifdef Py_TRACE_REFS\n",
    "    /* Define pointers to support a doubly-linked list of all live heap objects. */\n",
    "    #define _PyObject_HEAD_EXTRA            \\\n",
    "        struct _object *_ob_next;           \\\n",
    "        struct _object *_ob_prev;\n",
    "\n",
    "    #define _PyObject_EXTRA_INIT 0, 0,\n",
    "\n",
    "    #else\n",
    "    #define _PyObject_HEAD_EXTRA\n",
    "    #define _PyObject_EXTRA_INIT\n",
    "    #endif\n",
    "    \n",
    "\n",
    "如果 Py_TRACE_REFS 有定义，宏展开为两个指针，看名字是用来实现 双向链表 的：\n",
    "\n",
    "    struct _object *_ob_next;\n",
    "    struct _object *_ob_prev;\n",
    "    \n",
    "结合注释，双向链表用于跟踪所有**活跃堆对象**。\n",
    "\n",
    "对于**变长对象**，需要在PyObject基础上加入长度信息，这就是 PyVarObject ：\n",
    "\n",
    "    typedef struct {\n",
    "        PyObject ob_base;\n",
    "        Py_ssize_t ob_size; /* Number of items in variable part */\n",
    "    } PyVarObject;\n",
    "    \n",
    "变长对象比普通对象多一个字段 ob_size ，用于记录元素个数：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_01\\C1_03.png\" width=500 height=400>  \n",
    "\n",
    "\n",
    "至于具体对象，视其大小是否固定，需要包含头部 PyObject 或 PyVarObject 。 为此，头文件准备了两个宏定义，方便其他对象使用：\n",
    "\n",
    "    #define PyObject_HEAD          PyObject ob_base;\n",
    "    #define PyObject_VAR_HEAD      PyVarObject ob_base;\n",
    "    \n",
    "而对于大小不固定的 列表对象 ，则需要在 PyVarObject 头部基础上， 用一个动态数组加以实现，数组存储列表包含的对象，即 PyObject 指针：\n",
    "\n",
    "    typedef struct {\n",
    "        PyObject_VAR_HEAD\n",
    "        PyObject **ob_item;\n",
    "        Py_ssize_t allocated;\n",
    "    } PyListObject;\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_01\\C1_04.png\" width=500 height=400>  \n",
    "\n",
    "如图， PyListObject 底层由一个数组实现，关键字段是以下 3 个：\n",
    "\n",
    "- ob_item ，指向 动态数组 的指针，数组保存元素对象指针；\n",
    "- ob_size ，当前元素个数，即列表当前的 长度 ；\n",
    "- allocated ，动态数组总长度，即列表当前的 容量 ；"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### PyTypeObject，类型的基石"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "    typedef struct _typeobject {\n",
    "        PyObject_VAR_HEAD\n",
    "        const char *tp_name; /* For printing, in format \"<module>.<name>\" */\n",
    "        Py_ssize_t tp_basicsize, tp_itemsize; /* For allocation */\n",
    "\n",
    "        /* Methods to implement standard operations */\n",
    "        destructor tp_dealloc;\n",
    "        printfunc tp_print;\n",
    "\n",
    "        getattrfunc tp_getattr;\n",
    "        setattrfunc tp_setattr;\n",
    "\n",
    "        // ...\n",
    "        /* Attribute descriptor and subclassing stuff */\n",
    "        struct _typeobject *tp_base;\n",
    "\n",
    "        // ......\n",
    "    } PyTypeObject;\n",
    "    \n",
    "    \n",
    "可见 类型对象 PyTypeObject 是一个 变长对象 ，包含变长对象头部。 专有字段有：\n",
    "\n",
    "- **类型名称** ，即 tp_name 字段；\n",
    "- **类型的继承信息**，例如 tp_base 字段指向基类对象；\n",
    "- 创建实例对象时所需的**内存信息**，即 tp_basicsize 和 tp_itemsize 字段；\n",
    "- 该类型支持的相关**操作信息**，即 tp_print 、 tp_getattr 等函数指针；\n",
    "\n",
    "\n",
    "PyTypeObject 就是**类型对象**在 Python 中的表现形式，对应着面向对象中“类”的概念。 PyTypeObject 结构很复杂，但是我们不必在此刻完全弄懂它。 先有个大概的印象，知道 PyTypeObject 保存着对象的**元信息**，描述对象的**类型**即可。\n",
    "\n",
    "以 浮点 为例，考察 类型对象 和 实例对象 在内存中的形态和关系："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-25T04:59:04.138591Z",
     "start_time": "2021-03-25T04:59:04.123599Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'type'> True True\n"
     ]
    }
   ],
   "source": [
    "pi = 3.14\n",
    "e = 2.71\n",
    "print(float.__class__, type(pi) is float, type(e) is float)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "float 为浮点类型对象，系统中只有唯一一个，保存了所有浮点实例对象的元信息。 而浮点实例对象就有很多了，圆周率 pi 是一个，自然对数 e 是另一个，当然还有其他。\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_01\\C1_05.png\" width=500 height=400>  \n",
    "\n",
    "\n",
    "其中，两个浮点**实例对象**都是 PyFloatObject 结构体， 除了公共头部字段 ob_refcnt 和 ob_type ，专有字段 ob_fval 保存了对应的数值。 浮点**类型对象**是一个 PyTypeObject 结构体， 保存了类型名、内存分配信息以及浮点相关操作。 实例对象 ob_type 字段指向类型对象， Python 据此判断对象类型， 进而获悉关于对象的元信息，如操作方法等。 再次提一遍，float 、 pi 以及 e 等变量只是一个指向实际对象的指针。\n",
    "\n",
    "由于浮点 类型对象 全局唯一，在 C 语言层面作为一个全局变量静态定义即可，Python 的确就是这么做的。\n",
    "\n",
    "    PyTypeObject PyFloat_Type = {\n",
    "    PyVarObject_HEAD_INIT(&PyType_Type, 0) /* 第2行初始化ob_refcnt,ob_type 以及                                 * ob_size 三个字段；*/\n",
    "    \"float\",           /*  第3行将tp_name字段初始化成类型名称float */\n",
    "    sizeof(PyFloatObject),\n",
    "    0,\n",
    "    (destructor)float_dealloc,                  /* tp_dealloc */\n",
    "\n",
    "    // ...\n",
    "    (reprfunc)float_repr,                       /* tp_repr */\n",
    "\n",
    "    // ...\n",
    "    };\n",
    "    \n",
    " ob_type 指针指向 PyType_Type ，这也是一个静态定义的全局变量。 由此可见，代表\n",
    " **“类型的类型”** 即 type 的那个对象应该就是PyType_Type了"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### PyType_Type，类型的类型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "围观 PyFloat_Type 时，我们通过 ob_type 字段揪住了 PyType_Type 。 的确，它就是 type 的肉身。 PyType_Type 在 Object/typeobject.c 中定义：\n",
    "\n",
    "    PyTypeObject PyType_Type = {\n",
    "        PyVarObject_HEAD_INIT(&PyType_Type, 0)\n",
    "        \"type\",                                     /* tp_name */\n",
    "        sizeof(PyHeapTypeObject),                   /* tp_basicsize */\n",
    "        sizeof(PyMemberDef),                        /* tp_itemsize */\n",
    "        (destructor)type_dealloc,                   /* tp_dealloc */\n",
    "\n",
    "        // ...\n",
    "        (reprfunc)type_repr,                        /* tp_repr */\n",
    "\n",
    "        // ...\n",
    "    };\n",
    "    \n",
    "内建类型和自定义类对应的 PyTypeObject 对象都是这个通过 PyType_Type 创建的。 PyType_Type 在 Python 的类型机制中是一个至关重要的对象，它是所有类型的类型，称为 元类型 ( meta class )。 借助元类型，你可以实现很多神奇的高级操作。\n",
    "\n",
    "注意到， PyType_Type 将自己的 ob_type 字段设置成它自己(第 2 行)，这跟我们在Python 中看到的行为是吻合的："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-25T05:01:55.353972Z",
     "start_time": "2021-03-25T05:01:55.338978Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(type, type)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(type),type.__class__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "至此，元类型 type 在对象体系里的位置非常清晰了：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_01\\C1_06.png\" width=500 height=400>  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### PyBaseObject_Type，类型之基"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "object 是另一个特殊的类型，它是所有类型的基类。 那么，怎么找到它背后的实体呢？ 理论上，通过 PyFloat_Type 中 tp_base 字段顺藤摸瓜即可。PyBaseObject_Type 就是 object 背后的实体，先一睹其真容：\n",
    "\n",
    "    PyTypeObject PyBaseObject_Type = {\n",
    "        PyVarObject_HEAD_INIT(&PyType_Type, 0)\n",
    "        \"object\",                                   /* tp_name */\n",
    "        sizeof(PyObject),                           /* tp_basicsize */\n",
    "        0,                                          /* tp_itemsize */\n",
    "        object_dealloc,                             /* tp_dealloc */\n",
    "\n",
    "        // ...\n",
    "        object_repr,                                /* tp_repr */\n",
    "    };\n",
    "    \n",
    "注意到， ob_type 字段指向 PyType_Type 跟 object 在 Python 中的行为时相吻合的："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-25T05:10:57.602727Z",
     "start_time": "2021-03-25T05:10:57.585736Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(type, type)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "object.__class__, type(object)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "又注意到 PyType_Ready 函数初始化 PyBaseObject_Type 时，不设置 tp_base 字段。 因为继承链必须有一个终点，不然对象沿着继承链进行属性查找时便陷入死循环"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-25T05:07:57.543888Z",
     "start_time": "2021-03-25T05:07:57.527898Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None\n"
     ]
    }
   ],
   "source": [
    "print(object.__base__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-25T05:08:39.268264Z",
     "start_time": "2021-03-25T05:08:39.251286Z"
    },
    "hidden": true
   },
   "source": [
    "至此，我们完全弄清了 Python 对象体系中的所有实体以及关系，得到一幅完整的图画：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_01\\C1_07.png\" width=500 height=400>  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## 从创建到销毁，对象的生命周期"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "    static PyObject *\n",
    "    type_call(PyTypeObject *type, PyObject *args, PyObject *kwds)\n",
    "    {\n",
    "        PyObject *obj;\n",
    "\n",
    "        // ...\n",
    "        obj = type->tp_new(type, args, kwds); /* 调用类型对象 tp_new \n",
    "                                   * 函数指针申请内存 (第 7 行)；*/\n",
    "        obj = _Py_CheckFunctionResult((PyObject*)type, obj, NULL);\n",
    "        if (obj == NULL)\n",
    "            return NULL;\n",
    "\n",
    "        // ...\n",
    "        type = Py_TYPE(obj);\n",
    "        if (type->tp_init != NULL) {\n",
    "            int res = type->tp_init(obj, args, kwds); /* 必要时调用类型对象                       * tp_init 函数指针对对象进行 初始化 (第 15 行)；*/\n",
    "            if (res < 0) {\n",
    "                assert(PyErr_Occurred());\n",
    "                Py_DECREF(obj);\n",
    "                obj = NULL;\n",
    "            }\n",
    "            else {\n",
    "                assert(!PyErr_Occurred());\n",
    "            }\n",
    "        }\n",
    "        return obj;\n",
    "    }\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_01\\C1_08.png\" width=800 height=600>  \n",
    "\n",
    "总结一下，float 类型对象是 可调用对象 ，调用 float 即可创建实例对象：\n",
    "\n",
    "- 调用 float ， Python 最终执行其类型对象 type 的 tp_call 函数；\n",
    "- tp_call 函数调用 float 的 tp_new 函数为实例对象分配 内存空间 ；\n",
    "- tp_call 函数必要时进一步调用 tp_init 函数对实例对象进行 初始化 ；\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 对象的多态性"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Python 创建一个对象，比如 PyFloatObject ，会分配内存，并进行初始化。 此后， Python 内部统一通过一个 PyObject* 变量来保存和维护这个对象，而不是通过 PyFloatObject* 变量。\n",
    "\n",
    "通过 PyObject* 变量保存和维护对象，可以实现更抽象的上层逻辑，而不用关心对象的实际类型和实现细节。 \n",
    "\n",
    "    Py_hash_t\n",
    "    PyObject_Hash(PyObject *v)\n",
    "    {\n",
    "        PyTypeObject *tp = Py_TYPE(v);\n",
    "        if (tp->tp_hash != NULL)\n",
    "            return (*tp->tp_hash)(v);\n",
    "        /* To keep to the general practice that inheriting\n",
    "        * solely from object in C code should work without\n",
    "        * an explicit call to PyType_Ready, we implicitly call\n",
    "        * PyType_Ready here and then check the tp_hash slot again\n",
    "        */\n",
    "        if (tp->tp_dict == NULL) {\n",
    "            if (PyType_Ready(tp) < 0)\n",
    "                return -1;\n",
    "            if (tp->tp_hash != NULL)\n",
    "                return (*tp->tp_hash)(v);\n",
    "        }\n",
    "        /* Otherwise, the object can't be hashed */\n",
    "        return PyObject_HashNotImplemented(v);\n",
    "    }\n",
    "    \n",
    "函数先通过 ob_type 指针找到对象的类型 (第 4 行)； 然后通过类型对象的 tp_hash 函数指针，调用对应的哈希值计算函数 (第 6 行)。 换句话讲， PyObject_Hash 根据对象的类型，调用不同的函数版本。 这不就是**多态**吗？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 对象的行为"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "不同对象的行为不同，比如哈希值计算方法就不同，由类型对象中tp_hash字段决定。 除了 tp_hash ，我们看到PyTypeObject结构体还定义了很多函数指针，这些指针最终都会指向某个函数，或者为空。 这些函数指针可以看做是**类型对象**中定义的**操作**，这些操作决定对应**实例对象**在运行时的**行为**。\n",
    "\n",
    "尽管如此，不同对象也有一些共性。因此，以对象行为为依据，可以对对象进行分类：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_01\\C1_09.png\" width=800 height=600>  \n",
    "Python 便以此为依据，为每个类别都定义了一个**标准操作集**：\n",
    "- PyNumberMethods 结构体定义了**数值型**操作；\n",
    "- PySequenceMethods 结构体定义了**序列型**操作；\n",
    "- PyMappingMethods 结构体定义了**关联型**操作；\n",
    "只要**类型对象**提供相关**操作集**，**实例对象**便具备对应的**行为**。操作集字段如下：\n",
    "\n",
    "\n",
    "    typedef struct _typeobject {\n",
    "        PyObject_VAR_HEAD\n",
    "        const char *tp_name; /* For printing, in format \"<module>.<name>\" */\n",
    "        Py_ssize_t tp_basicsize, tp_itemsize; /* For allocation */\n",
    "\n",
    "        // ...\n",
    "        /* Method suites for standard classes */\n",
    "\n",
    "        PyNumberMethods *tp_as_number;\n",
    "        PySequenceMethods *tp_as_sequence;\n",
    "        PyMappingMethods *tp_as_mapping;\n",
    "\n",
    "        // ...\n",
    "        /* Functions to access object as input/output buffer */\n",
    "        PyBufferProcs *tp_as_buffer;\n",
    "\n",
    "        // ...\n",
    "        } PyTypeObject;\n",
    "        \n",
    "以 float 为例，类型对象 PyFloat_Type 相关字段是这样初始化的：\n",
    "\n",
    "    static PyNumberMethods float_as_number = {\n",
    "    float_add,          /* nb_add */\n",
    "    float_sub,          /* nb_subtract */\n",
    "    float_mul,          /* nb_multiply */\n",
    "    float_rem,          /* nb_remainder */\n",
    "    float_divmod,       /* nb_divmod */\n",
    "    float_pow,          /* nb_power */\n",
    "    // ...\n",
    "    };\n",
    "\n",
    "    PyTypeObject PyFloat_Type = {\n",
    "        PyVarObject_HEAD_INIT(&PyType_Type, 0)\n",
    "        \"float\",\n",
    "        sizeof(PyFloatObject),\n",
    "\n",
    "        // ...\n",
    "        &float_as_number,                           /* tp_as_number */\n",
    "        0,                                          /* tp_as_sequence */\n",
    "        0,                                          /* tp_as_mapping */\n",
    "\n",
    "        // ...\n",
    "    };\n",
    "    \n",
    "- 字段 tp_as_number 非空，因此 float 对象 支持数值型操作 ；\n",
    "- 字段 tp_as_sequence 为空，因此 float 对象 不支持序列型操作 ；\n",
    "- 字段 tp_as_mapping 为空，因此 float 对象 不支持关联型操作 ；"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 引用计数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-25T05:41:57.476468Z",
     "start_time": "2021-03-25T05:41:57.465467Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sys\n",
    "a = 3.14  # Python 对象创建后，引用计数设为 1 \n",
    "sys.getrefcount(a)# 对象作为函数参数传递，需要将引用计数加一，避免对象被提前销毁；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-25T05:41:58.473512Z",
     "start_time": "2021-03-25T05:41:58.460505Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = a \n",
    "sys.getrefcount(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-25T05:42:35.533313Z",
     "start_time": "2021-03-25T05:42:35.521321Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3.14]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l  = [a]\n",
    "print(l)\n",
    "sys.getrefcount(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-25T05:42:53.528847Z",
     "start_time": "2021-03-25T05:42:53.510858Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "del b\n",
    "sys.getrefcount(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-25T05:43:16.359008Z",
     "start_time": "2021-03-25T05:43:16.349999Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l.clear()\n",
    "sys.getrefcount(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-25T05:43:24.624079Z",
     "start_time": "2021-03-25T05:43:24.620096Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "del a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "在 Python 中，很多场景都涉及引用计数的调整，例如：\n",
    "\n",
    "- 容器操作；\n",
    "- 变量赋值；\n",
    "- 函数参数传递；\n",
    "- 属性操作；\n",
    "\n",
    "为此， Python 定义了两个非常重要的宏，用于维护对象应用计数。 其中， Py_INCREF 将对象应用计数加一 ( 3 行)：\n",
    "\n",
    "    #define Py_INCREF(op) (                         \\\n",
    "    _Py_INC_REFTOTAL  _Py_REF_DEBUG_COMMA       \\\n",
    "    ((PyObject *)(op))->ob_refcnt++)\n",
    "Py_DECREF 将引用计数减一 ( 5 行)，并在引用计数为 0 是回收对象 ( 8 行)：\n",
    "\n",
    "    #define Py_DECREF(op)                                   \\\n",
    "        do {                                                \\\n",
    "            PyObject *_py_decref_tmp = (PyObject *)(op);    \\\n",
    "            if (_Py_DEC_REFTOTAL  _Py_REF_DEBUG_COMMA)       \\\n",
    "            --(_py_decref_tmp)->ob_refcnt != 0)             \\\n",
    "                _Py_CHECK_REFCNT(_py_decref_tmp)            \\\n",
    "            else                                            \\\n",
    "                _Py_Dealloc(_py_decref_tmp);                \\\n",
    "        } while (0)\n",
    "        \n",
    "当一个对象引用计数为 0 ， Python 便调用对象对应的析构函数销毁对象，但这并不意味着对象内存一定会回收。 为了提高内存分配效率， Python 为一些常用对象维护了内存池， 对象回收后内存进入内存池中，以便下次使用，由此 避免**频繁申请、释放内存** 。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# 内建对象"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## 小试牛刀，解剖浮点对象 float"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "    PyTypeObject PyFloat_Type = {\n",
    "        PyVarObject_HEAD_INIT(&PyType_Type, 0)\n",
    "        \"float\",\n",
    "        sizeof(PyFloatObject),\n",
    "        0,\n",
    "        (destructor)float_dealloc,                  /* tp_dealloc */\n",
    "        0,                                          /* tp_print */\n",
    "        0,                                          /* tp_getattr */\n",
    "        0,                                          /* tp_setattr */\n",
    "        0,                                          /* tp_reserved */\n",
    "        (reprfunc)float_repr,                       /* tp_repr */\n",
    "        &float_as_number,                           /* tp_as_number */\n",
    "        0,                                          /* tp_as_sequence */\n",
    "        0,                                          /* tp_as_mapping */\n",
    "        (hashfunc)float_hash,                       /* tp_hash */\n",
    "        0,                                          /* tp_call */\n",
    "        (reprfunc)float_repr,                       /* tp_str */\n",
    "        PyObject_GenericGetAttr,                    /* tp_getattro */\n",
    "        0,                                          /* tp_setattro */\n",
    "        0,                                          /* tp_as_buffer */\n",
    "        Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,   /* tp_flags */\n",
    "        float_new__doc__,                           /* tp_doc */\n",
    "        0,                                          /* tp_traverse */\n",
    "        0,                                          /* tp_clear */\n",
    "        float_richcompare,                          /* tp_richcompare */\n",
    "        0,                                          /* tp_weaklistoffset */\n",
    "        0,                                          /* tp_iter */\n",
    "        0,                                          /* tp_iternext */\n",
    "        float_methods,                              /* tp_methods */\n",
    "        0,                                          /* tp_members */\n",
    "        float_getset,                               /* tp_getset */\n",
    "        0,                                          /* tp_base */\n",
    "        0,                                          /* tp_dict */\n",
    "        0,                                          /* tp_descr_get */\n",
    "        0,                                          /* tp_descr_set */\n",
    "        0,                                          /* tp_dictoffset */\n",
    "        0,                                          /* tp_init */\n",
    "        0,                                          /* tp_alloc */\n",
    "        float_new,                                  /* tp_new */\n",
    "    };\n",
    "    \n",
    "PyFloat_Type 中保存了很多关于浮点对象的 元信息 ，关键字段包括：\n",
    "\n",
    "- tp_name 字段保存类型名称，常量 float ；\n",
    "- tp_dealloc 、 tp_init 、 tp_alloc 和 tp_new 字段是对象创建销毁相关函数；\n",
    "- tp_repr 字段是生成语法字符串表示形式的函数；\n",
    "- tp_str 字段是生成普通字符串表示形式的函数；\n",
    "- tp_as_number 字段是数值操作集；\n",
    "- tp_hash 字段是哈希值生成函数；\n",
    "\n",
    "PyFloat_Type 很重要，作为浮点**类型对象**，它决定了浮点**实例对象**的**生死和行为**。 接下来，我们以不同小节分别展开讨论。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 对象的创建"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "<img style=\"float: middle;\" src=\".\\imgs\\C_01\\C1_08.png\" width=800 height=600>  \n",
    "\n",
    "创建对象的过程\n",
    "\n",
    "注意到， float 类型对象 PyFloat_Type 中 tp_init 函数指针为空。 这是因为 float 是一种很简单的对象，初始化操作只需要一个赋值语句，在 tp_new 中完成即可。\n",
    "\n",
    "除了通用的流程， Python 为内置对象实现了对象创建 API ，简化调用，提高效率：\n",
    "\n",
    "    PyObject *\n",
    "    PyFloat_FromDouble(double fval);\n",
    "\n",
    "    PyObject *\n",
    "    PyFloat_FromString(PyObject *v);\n",
    "    \n",
    " - PyFloat_FromDouble 函数通过浮点值创建浮点对象；\n",
    " - PyFloat_FromString 函数通过字符串对象创建浮点对象；\n",
    " \n",
    " 以 PyFloat_FromDouble 为例，特化的对象创建流程如下：\n",
    " \n",
    " \n",
    "    PyObject *\n",
    "    PyFloat_FromDouble(double fval)\n",
    "    {\n",
    "    PyFloatObject *op = free_list;\n",
    "    if (op != NULL) {\n",
    "        free_list = (PyFloatObject *) Py_TYPE(op);\n",
    "        numfree--;\n",
    "    } else {\n",
    "        op = (PyFloatObject*) PyObject_MALLOC(sizeof(PyFloatObject));\n",
    "        if (!op)\n",
    "            return PyErr_NoMemory();\n",
    "    }\n",
    "    /* Inline PyObject_New */\n",
    "    (void)PyObject_INIT(op, &PyFloat_Type);\n",
    "    op->ob_fval = fval;\n",
    "    return (PyObject *) op;\n",
    "    }\n",
    "\n",
    "1. 为对象**分配内存空间**，优先使用空闲对象缓存池 (第 4-12 行)；\n",
    "2. 初始化**对象类型**字段 ob_type以及**引用计数**字段ob_refcnt (第 14 行)；\n",
    "3. 将 ob_fval 字段初始化为指定**浮点值**(第 15 行)；\n",
    "\n",
    "其中，宏 PyObject_INIT 在头文件 Include/objimpl.h 中定义：\n",
    "\n",
    "    #define PyObject_INIT(op, typeobj) \\\n",
    "    ( Py_TYPE(op) = (typeobj), _Py_NewReference((PyObject *)(op)), (op) )\n",
    "    \n",
    "宏 _Py_NewReference 将对象引用计数初始化为 1 ，在 Include/Object.h 中定义：\n",
    "\n",
    "    #define _Py_NewReference(op) (                          \\\n",
    "    _Py_INC_TPALLOCS(op) _Py_COUNT_ALLOCS_COMMA         \\\n",
    "    _Py_INC_REFTOTAL  _Py_REF_DEBUG_COMMA               \\\n",
    "    Py_REFCNT(op) = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### 对象的销毁"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "当对象不再需要时， Python 通过 Py_DECREF 或者 Py_XDECREF 宏减少引用计数； 当引用计数降为 0 时， Python 通过 _Py_Dealloc 宏回收对象。\n",
    "\n",
    "_Py_Dealloc 宏调用类型对象 PyFloat_Type 中的 tp_dealloc 函数指针：\n",
    "\n",
    "    #define _Py_Dealloc(op) (                               \\\n",
    "    _Py_INC_TPFREES(op) _Py_COUNT_ALLOCS_COMMA          \\\n",
    "    (*Py_TYPE(op)->tp_dealloc)((PyObject *)(op)))\n",
    "\n",
    "因此，实际调用的函数是 float_dealloc (代码在下一小节 空闲对象缓存池 中解析)：\n",
    "\n",
    "    static void\n",
    "    float_dealloc(PyFloatObject *op)\n",
    "    {\n",
    "        if (PyFloat_CheckExact(op)) {\n",
    "            if (numfree >= PyFloat_MAXFREELIST)  {\n",
    "                PyObject_FREE(op);\n",
    "                return;\n",
    "            }\n",
    "            numfree++;\n",
    "            Py_TYPE(op) = (struct _typeobject *)free_list;\n",
    "            free_list = op;\n",
    "        }\n",
    "        else\n",
    "            Py_TYPE(op)->tp_free((PyObject *)op);\n",
    "    }\n",
    "    \n",
    "总结起来，对象从创建到销毁整个生命周期所涉及的 关键函数、宏及调用关系 如下：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_02.png\" width=800 height=600>  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### 空闲对象缓存池"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "浮点运算背后涉及**大量临时对象创建以及销毁**，以计算圆周率为例：\n",
    "\n",
    "    area = pi * r ** 2\n",
    "    \n",
    "这个语句首先计算半径 r 的平方，中间结果由一个临时对象来保存，假设是 t ； 然后计算圆周率 pi 与 t 的乘积，得到最终结果并赋值给变量 area ； 最后，销毁临时对象 t 。 这么简单的语句，背后居然都隐藏着一个临时对象的创建以及销毁操作！\n",
    "\n",
    "创建对象时需要分配内存，销毁对象时又需要回收内存。 大量临时对象创建销毁 ，意味着**大量内存分配回收操作*8，这显然是是不可接受的。\n",
    "\n",
    "因此 Python 在浮点对象销毁后，并不急于回收内存，而是将对象放入一个 **空闲链表**。 后续需要创建浮点对象时，先到空闲链表中取，省去分配内存的开销。\n",
    "\n",
    "浮点对象的空闲链表同样在 Objects/floatobject.c 中定义：\n",
    "\n",
    "    #ifndef PyFloat_MAXFREELIST\n",
    "    #define PyFloat_MAXFREELIST    100\n",
    "    #endif\n",
    "    static int numfree = 0;\n",
    "    static PyFloatObject *free_list = NULL;\n",
    "    \n",
    "- free_list 变量，指向空闲链表 头节点 的指针；\n",
    "- numfree 变量，维护空闲链表 当前长度 ；\n",
    "- PyFloat_MAXFREELIST 宏，限制空闲链表的 最大长度 ，避免占用过多内存；\n",
    "\n",
    "为了保持简洁， Python 把 ob_type 字段当作 next 指针来用，将空闲对象串成链表：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_03.png\" width=800 height=600>  \n",
    "\n",
    "由此一来，需要创建浮点对象时，可以从链表中取出空闲对象，省去申请内存的开销！ 以 PyFloat_FromDouble 为例：\n",
    "\n",
    "    PyFloatObject *op = free_list;\n",
    "    if (op != NULL) {\n",
    "        free_list = (PyFloatObject *) Py_TYPE(op);\n",
    "        numfree--;\n",
    "    } else {\n",
    "        op = (PyFloatObject*) PyObject_MALLOC(sizeof(PyFloatObject));\n",
    "        // ...\n",
    "    }\n",
    "    \n",
    "1. 检查 free_list 是否为空(第 2 行)；\n",
    "2. 如果 free_list 非空，取出头节点备用， 并将 numfree 减一(第 3-4 行)；\n",
    "3. 如果 free_list 为空，则调用 PyObject_MALLOC 分配内存(第 6 行)；\n",
    "\n",
    "对象销毁时， Python 将其缓存在空闲链表中，以备后用。考察 float_dealloc 函数：\n",
    "\n",
    "    if (numfree >= PyFloat_MAXFREELIST)  {\n",
    "        PyObject_FREE(op);\n",
    "        return;\n",
    "    }\n",
    "    numfree++;\n",
    "    Py_TYPE(op) = (struct _typeobject *)free_list;\n",
    "    free_list = op;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### 对象的行为`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "PyFloat_Type 中定义了很多函数指针，包括 tp_repr 、 tp_str 、 tp_hash 等。 这些函数指针将一起决定 float 对象的行为。\n",
    "**数值操作集**\n",
    "由于加减乘除等数值操作很常见， Python 将其抽象成数值操作集 PyNumberMethods 。 数值操作集 PyNumberMethods 在头文件 Include/object.h 中定义：\n",
    "\n",
    "    typedef struct {\n",
    "    /* Number implementations must check *both*\n",
    "    arguments for proper type and implement the necessary conversions\n",
    "    in the slot functions themselves. */\n",
    "\n",
    "    binaryfunc nb_add;\n",
    "    binaryfunc nb_subtract;\n",
    "    binaryfunc nb_multiply;\n",
    "    binaryfunc nb_remainder;\n",
    "    binaryfunc nb_divmod;\n",
    "    ternaryfunc nb_power;\n",
    "    unaryfunc nb_negative;\n",
    "    // ...\n",
    "\n",
    "    binaryfunc nb_inplace_add;\n",
    "    binaryfunc nb_inplace_subtract;\n",
    "    binaryfunc nb_inplace_multiply;\n",
    "    binaryfunc nb_inplace_remainder;\n",
    "    ternaryfunc nb_inplace_power;\n",
    "    //...\n",
    "    } PyNumberMethods;\n",
    "    \n",
    "PyNumberMethods 定义了各种数学算子的处理函数，数值计算最终由这些函数执行。 处理函数根据参数个数可以分为： 一元函数 ( unaryfunc )、 二元函数 ( binaryfunc )等。\n",
    "\n",
    "回到 Objects/floatobject.c 观察浮点对象数值操作集 float_as_number 是如何初始化的：\n",
    "\n",
    "    static PyNumberMethods float_as_number = {\n",
    "        float_add,          /* nb_add */\n",
    "        float_sub,          /* nb_subtract */\n",
    "        float_mul,          /* nb_multiply */\n",
    "        float_rem,          /* nb_remainder */\n",
    "        float_divmod,       /* nb_divmod */\n",
    "        float_pow,          /* nb_power */\n",
    "        (unaryfunc)float_neg, /* nb_negative */\n",
    "        // ...\n",
    "\n",
    "        0,                  /* nb_inplace_add */\n",
    "        0,                  /* nb_inplace_subtract */\n",
    "        0,                  /* nb_inplace_multiply */\n",
    "        0,                  /* nb_inplace_remainder */\n",
    "        0,                  /* nb_inplace_power */\n",
    "        // ...\n",
    "    };\n",
    "    \n",
    "以加法为例，以下语句在 Python 内部最终由 float_add 函数执行：\n",
    "\n",
    "    >>> a = 1.5\n",
    "    >>> b = 1.1\n",
    "    >>> a + b\n",
    "    2.6\n",
    "    \n",
    "float_add 是一个 二元函数 ，同样位于 Objects/floatobject.h 中：\n",
    "\n",
    "    static PyObject *\n",
    "    float_add(PyObject *v, PyObject *w)\n",
    "    {\n",
    "        double a,b;\n",
    "        CONVERT_TO_DOUBLE(v, a);\n",
    "        CONVERT_TO_DOUBLE(w, b);\n",
    "        PyFPE_START_PROTECT(\"add\", return 0)\n",
    "        a = a + b;\n",
    "        PyFPE_END_PROTECT(a)\n",
    "        return PyFloat_FromDouble(a);\n",
    "    }\n",
    "    \n",
    "函数实现只有寥寥几步： 首先，将两个参数对象转化成浮点值( 5-6 行)； 然后，对两个浮点值求和( 8 行)； 最后，创建一个新浮点对象保存计算结果并返回( 10 行)。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "##  int 对象，永不溢出的整数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### int 对象的设计"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "int 对象在 Include/longobject.h 头文件中定义：\n",
    "\n",
    "    typedef struct _longobject PyLongObject; /* Revealed in longintrepr.h */\n",
    "    \n",
    "我们顺着注释找到了 Include/longintrepr.h ，实现 int 对象的结构体真正藏身之处：\n",
    "\n",
    "    struct _longobject {\n",
    "        PyObject_VAR_HEAD\n",
    "        digit ob_digit[1];\n",
    "    };\n",
    "\n",
    "这个结构我们并不陌生，说明 int 对象是一个变长对象。除了变长对象都具有的公共头部，还有一个 digit 数组，整数值应该就存储在这个数组里面。 digit 又是什么呢？同样在 Include/longintrepr.h 头文件，我们找到它的定义：\n",
    "    \n",
    "    #if PYLONG_BITS_IN_DIGIT == 30\n",
    "    typedef uint32_t digit;\n",
    "    // ...\n",
    "    #elif PYLONG_BITS_IN_DIGIT == 15\n",
    "    typedef unsigned short digit;\n",
    "    // ...\n",
    "    #endif\n",
    "    \n",
    "看上去 digit 就是一个 C 语言整数，至此我们知晓 int 对象是通过整数数组来实现大整数的。一个 C 整数类型不够就两个嘛，两个不够那就 n 个！至于整数数组用什么整数类型来实现， Python 提供了两个版本，一个是 32 位的 uint32_t ，一个是 16 位的 unsigned short ，编译 Python 解析器时可以通过宏定义指定选用的版本。\n",
    "\n",
    "Python 作者为什么要这样设计呢？这主要是出于内存方面的考量：对于范围不大的整数，用 16 位整数表示即可，用 32 位就有点浪费。本人却觉得由于整数对象公共头部已经占了 24 字节，省这 2 个字节其实意义不大。\n",
    "\n",
    "\n",
    "|整数对象|对象大小（16 位）|对象大小（32 位）|\n",
    "|  :-:  |:-:|:-:|\n",
    "|1|\t24 + 2 * 1 = 26\t|24 + 4 * 1 = 28|\n",
    "|1000000|\t24 + 2 * 2 = 28\t|24 + 4 * 1 = 28|\n",
    "|10000000000|\t24 + 2 * 3 = 30\t|24 + 4 * 2 = 32|\n",
    "\n",
    "由此可见，选用 16 位整数数组时， int 对象内存增长的粒度更小，有些情况下可以节省 2 个字节。但是这 2 字节相比 24 字节的变长对象公共头部显得微不足道，因此 Python 默认选用 32 位整数数组也就不奇怪了。\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_04.png\" width=800 height=600>  \n",
    "\n",
    "如上图，对于比较大的整数， Python 将其拆成若干部分，保存在 ob_digit 数组中。然而我们注意到在结构体定义中， ob_digit 数组长度却固定为 1 ，这是为什么呢？由于 C 语言中数组长度不是类型信息，我们可以根据实际需要为 ob_digit 数组分配足够的内存，并将其当成长度为 n 的数组操作。这也是 C 语言中一个常用的编程技巧。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 大整数布局"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-25T07:16:20.458127Z",
     "start_time": "2021-03-25T07:16:20.450117Z"
    },
    "hidden": true
   },
   "source": [
    "整数分为 正数 、 负数 和 零 ， Python 规定不同整数在 int 对象中的存储方式，要点可以总结为 3 条：\n",
    "\n",
    "- 整数 绝对值 根据实际情况分为若干部分，保存于 ob_digit 数组中；\n",
    "- ob_digit 数组长度 保存于 ob_size 字段，对于 负整数 的情况，ob_size 为负；\n",
    "- 整数 零 以 ob_size 等于 0 来表示，ob_digit 数组为空；\n",
    "\n",
    "接下来，我们以 5 个典型的例子详细介绍这几条规则：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_05.png\" width=800 height=600>  \n",
    "1. 对于整数 0 ， ob_size 字段等于 0 ， ob_digit 数组为空，无需分配。\n",
    "2. 对于整数 10 ，其绝对值保存于 ob_digit 数组中，数组长度为 1 ， ob_size 字段等于 1 。\n",
    "3. 对于整数 -10 ，其绝对值同样保存于 ob_digit 数组中，但由于 -10 为负数， ob_size 字段等于 -1 。\n",
    "4. 对于整数 1073741824 ( 2 的 30 次方)，由于 Python 只使用 32 整数的后 30 位，需要另一个整数才能存储，整数数组长度为 2 。绝对值这样计算：$2^{30}*1+2^0*0=10737418242$ \n",
    "5. 对于整数 -4294967297 (负的 2 的 32 次方加 1 )，同样要长度为 2 的 ob_digit 数组，但 ob_size 字段为负。绝对值这样计算：$2^{30}*4+2^0*1=42949672972$\n",
    "\n",
    "至于为什么 Python 只用 ob_digit 数组整数的后 30 位，其实跟加法进位有关。如果全部 32 位都用来保存绝对值，那么为了保证加法不溢出 (产生进位)，需要先强制转换成 64 位类型后在进行计算。但牺牲最高 1 位后，加法运算便不用担心进位溢出了。那么，为什么 Python 牺牲最高 2 位呢？我猜这是为了和 16 位整数方案统一起来：如果选用 16 位整数作为数组， Python 则只使用其中 15 位。\n",
    "\n",
    "由于篇幅关系，大整数 数值运算 留在下节详细介绍。届时，我们将深入源码，体验大整数运算的精妙之处。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 小整数静态对象池"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "通过前面章节的学习，我们知道整数对象是**不可变对象**，整数运算结果是以**新对象**返回的："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-25T07:22:31.182248Z",
     "start_time": "2021-03-25T07:22:31.173239Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "140710621583008\n",
      "140710621583040\n"
     ]
    }
   ],
   "source": [
    "a = 1\n",
    "print(id(a))\n",
    "a+= 1\n",
    "print(id(a))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Python 这样的设计带来一个性能缺陷，程序运行时势必有大量对象创建销毁。创建对象需要分配内存，对象销毁需要将内存回收，严重影响性能。编写一个循环 100 次的循环，便需要创建 100 个 int 对象。这显然是难以接受的。 Python 的解决方案是：预先将常用的整数对象创建好，以备后用，这就是**小整数对象池**。小整数对象池在 Objects/longobject.c 中实现，关键代码如下：\n",
    "\n",
    "    #ifndef NSMALLPOSINTS\n",
    "    #define NSMALLPOSINTS           257\n",
    "    #endif\n",
    "    #ifndef NSMALLNEGINTS\n",
    "    #define NSMALLNEGINTS           5\n",
    "    #endif\n",
    "\n",
    "    static PyLongObject small_ints[NSMALLNEGINTS + NSMALLPOSINTS];\n",
    "    \n",
    "- NSMALLPOSINTS 宏规定了对象池 正数个数 (从 0 开始，包括 0 )，默认 257 个；\n",
    "- NSMALLNEGINTS 宏规定了对象池 负数个数 ，默认 5 个；\n",
    "- small_ints 是一个整数对象数组，保存预先创建好的小整数对象；\n",
    "\n",
    "以默认配置为例， Python 启动后静态创建一个包含 262 个元素的整数数组并依次初始化为 -5 到 -1 这 5 个负数、零以及 1 到 256 这 256 个正数。 小整数对象池 结构如下：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_06.png\" width=800 height=600>  \n",
    "\n",
    "至于为什么选择静态缓存从 -5 到 256 之间的小整数，主要是出于某种**权衡**：这个范围内的整数使用**频率**很高 ，而缓存这些小整数的**内存开销相对可控**。很多程序开发场景都没有固定的正确答案，需要根据实际情况平衡利弊"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 总结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "与主流编程语言相比， Python 中的整数 永远不会溢出 ，应用起来非常省心。 Python 的整数对象是 变长对象 ，能够按需串联多个 C 整数类型，实现大整数表示。整数对象关键字段包括 底层整数数组 ob_digit 以及 数组长度 ob_size 。整数数值按照以下规则保存：\n",
    "- 整数 绝对值 拆分成多个部分，存放于 底层整数数组 ob_digit ；\n",
    "- 底层数组长度保存在 ob_size 字段，如果整数为负， ob_size 也为负；\n",
    "- 对于整数 0 ，底层数组为空， ob_size 字段为 0 ；\n",
    "\n",
    "由于整数对象是**不可变对象**，任何整数运算结果都以新对象返回，而对象创建销毁开销却不小。为了优化整数对象的性能， Python在启动时将使用**频率较高**的小整数预先创建好，这就是**小整数缓存池**。默认情况下，小整数缓存池缓存从 -5 到 256 之间的整数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## int 源码解析：如何实现大整数运算？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Python 整数对象通过串联多个 C 整数类型，实现大整数的表示。整数对象内部包含一个 C 整数数组，数组长度与对象表示的数值大小相关，因此整数对象也是**变长对象**。深入源码细节前，我们先重温整数对象的内部结构：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_04.png\" width=800 height=600>  \n",
    "- ob_digit 为 C 整数数组，用于存储被保存整数的 绝对值 ；\n",
    "- ob_size 为 变长对象 关键字段，维护数组长度以及被保存整数的 符号 ；\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_05.png\" width=800 height=600>  \n",
    "用整数数组实现大整数的思路其实平白无奇，难点在于大整数**数学运算**的实现，这是也比较考验编程功底的地方。不管是校招还是社招面试，大整数实现都是一个比较常见的考察点，必须掌握。接下来，我们继续深入整数对象源码( Objects/longobject.c )，窥探大整数运算的秘密。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 数学运算概述"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "根据我们在 对象模型 中学到的知识，对象的行为由对象的 类型 决定。因此，整数对象数学运算的秘密藏在整数类型对象中。我们在 Objects/longobject.c 中找到整数类型对象( PyLong_Type )，其定义如下所示：\n",
    "\n",
    "    PyTypeObject PyLong_Type = {\n",
    "        PyVarObject_HEAD_INIT(&PyType_Type, 0)\n",
    "        \"int\",                                      /* tp_name */\n",
    "        offsetof(PyLongObject, ob_digit),           /* tp_basicsize */\n",
    "        sizeof(digit),                              /* tp_itemsize */\n",
    "        long_dealloc,                               /* tp_dealloc */\n",
    "\n",
    "        //...\n",
    "\n",
    "        &long_as_number,                            /* tp_as_number */\n",
    "\n",
    "        //...\n",
    "\n",
    "        long_new,                                   /* tp_new */\n",
    "        PyObject_Del,                               /* tp_free */\n",
    "    };\n",
    "    \n",
    "类型对象中， tp_as_number 是一个关键字段。该字段指向一个 PyNumberMethods 结构体，结构体保存了各种数学运算的 函数指针 。我们顺藤摸瓜，很快便找到整数对象所有数学运算的处理函数：\n",
    "\n",
    "    static PyNumberMethods long_as_number = {\n",
    "        (binaryfunc)long_add,       /*nb_add*/\n",
    "        (binaryfunc)long_sub,       /*nb_subtract*/\n",
    "        (binaryfunc)long_mul,       /*nb_multiply*/\n",
    "        long_mod,                   /*nb_remainder*/\n",
    "        long_divmod,                /*nb_divmod*/\n",
    "        long_pow,                   /*nb_power*/\n",
    "        (unaryfunc)long_neg,        /*nb_negative*/\n",
    "        (unaryfunc)long_long,       /*tp_positive*/\n",
    "        (unaryfunc)long_abs,        /*tp_absolute*/\n",
    "        (inquiry)long_bool,         /*tp_bool*/\n",
    "        (unaryfunc)long_invert,     /*nb_invert*/\n",
    "        long_lshift,                /*nb_lshift*/\n",
    "        (binaryfunc)long_rshift,    /*nb_rshift*/\n",
    "        long_and,                   /*nb_and*/\n",
    "        long_xor,                   /*nb_xor*/\n",
    "        long_or,                    /*nb_or*/\n",
    "        long_long,                  /*nb_int*/\n",
    "        // ...\n",
    "    };\n",
    "至此，我们明确了整数对象支持的全部 数学运算 ，以及对应的 处理函数 (下表仅列举常用部分)：\n",
    "\n",
    "|数学运算|处理函数|示例|\n",
    "| :-: | :-: | :-: |\n",
    "|加法|long_add|a + b|\n",
    "|减法|long_sub|a - b|\n",
    "|乘法|long_mul|a * b|\n",
    "|取模|long_mod|a % b|\n",
    "|除法|long_divmod|a / b|\n",
    "|指数|long_pow|a ** b|\n",
    "\n",
    "最后，我们用一张图片来总结 整数对象 、 整数类型对象 以及 整数数学运算处理函数 之间的关系：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_07.png\" width=800 height=600>  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 加法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "如何为一个由数组表示的大整数实现加法？问题答案得在 long_add 函数中找，该函数是整数对象**加法处理函数** 。我们再接再厉，扒开 long_add 函数看个究竟(同样位于 Objects/longobject.c )：\n",
    "\n",
    "    static PyObject *\n",
    "    long_add(PyLongObject *a, PyLongObject *b)\n",
    "    {\n",
    "    PyLongObject *z;\n",
    "\n",
    "    CHECK_BINOP(a, b);\n",
    "\n",
    "    if (Py_ABS(Py_SIZE(a)) <= 1 && Py_ABS(Py_SIZE(b)) <= 1) {\n",
    "        return PyLong_FromLong(MEDIUM_VALUE(a) + MEDIUM_VALUE(b));\n",
    "    }\n",
    "    if (Py_SIZE(a) < 0) {\n",
    "        if (Py_SIZE(b) < 0) {\n",
    "            z = x_add(a, b);\n",
    "            if (z != NULL) {\n",
    "                assert(Py_REFCNT(z) == 1);\n",
    "                Py_SIZE(z) = -(Py_SIZE(z));\n",
    "            }\n",
    "        }\n",
    "        else\n",
    "            z = x_sub(b, a);\n",
    "    }\n",
    "    else {\n",
    "        if (Py_SIZE(b) < 0)\n",
    "            z = x_sub(a, b);\n",
    "        else\n",
    "            z = x_add(a, b);\n",
    "    }\n",
    "    return (PyObject *)z;\n",
    "    }\n",
    "    \n",
    "long_add 函数并不长，调用其他辅助函数完成加法运算，主体逻辑如下：\n",
    "\n",
    "- 第 4 行，定义变量 z 用于临时保存计算结果；\n",
    "- 第 8-10 行，如果两个对象数组长度均不超过 1 ， 用 MEDIUM_VALUE 宏将其转化成 C 整数进行运算即可；\n",
    "- 第 13-17 行，如果两个整数均为 负数 ，调用 x_add 计算两者绝对值之和，再将结果符号设置为负( 16 行处)；\n",
    "- 第 20 行，如果 a 为负数， b 为正数，调用 x_sub 计算 b 和 a 的绝对值之差即为最终结果；\n",
    "- 第 24 行，如果 a 为正数， b 为负数，调用 x_sub 计算 a 和 b 的绝对值之差即为最终结果；\n",
    "- 第 26 行，如果两个整数均为正数，调用 x_add 计算两个绝对值之和即为最终结果；\n",
    "\n",
    "最后 4 个步骤看起来很复杂，也有点令人费解。别担心，这只是初中数学的基本知识：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_08.png\" width=500 height=400>  \n",
    "\n",
    "因此， long_add 函数将整数加法转换成 绝对值加法 ( x_add )以及 绝对值 减法( x_sub )：\n",
    "\n",
    "- x_add(a, b) ，计算两者绝对值之和，即 |a|+|b| ；\n",
    "- x_sub(a, b) ，计算两者绝对值之差，即 |a|-|b| ；\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_09.png\" width=500 height=400>  \n",
    "\n",
    "由于绝对值加、减法不用考虑符号对计算结果的影响，实现更为简单，这是 Python 将整数运算转化成绝对值运算的缘由。虽然我们还没弄明白 x_add 和 x_sub 的细节，但也能从中体会到程序设计中逻辑**划分**与**组合**的艺术。优秀的代码真的很美！\n",
    "\n",
    "你可能有些疑问：大整数运算这么复杂，性能一定很差吧？这是必然的，整数数值越大，整数对象底层数组越长，运算开销也就越大。好在运算处理函数均以快速通道对小整数运算进行优化，将额外开销降到最低。\n",
    "\n",
    "以 long_add 为例， 8-10 行便是一个快速通道：如果参与运算的整数对象底层数组长度均不超过 1 ，直接将整数对象转化成 C 整数类型进行运算，性能损耗极小。满足这个条件的整数范围在 -1073741823~1073741823 之间，足以覆盖程序运行时的绝大部分运算场景。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 绝对值加法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "x_add 用于计算两个整数对象绝对值之和，源码同样位于 Objects/longobject.c ：\n",
    "\n",
    "    static PyLongObject *\n",
    "    x_add(PyLongObject *a, PyLongObject *b)\n",
    "    {\n",
    "        Py_ssize_t size_a = Py_ABS(Py_SIZE(a)), size_b = Py_ABS(Py_SIZE(b));\n",
    "        PyLongObject *z;\n",
    "        Py_ssize_t i;\n",
    "        digit carry = 0;\n",
    "\n",
    "        /* Ensure a is the larger of the two: */\n",
    "        if (size_a < size_b) {\n",
    "            { PyLongObject *temp = a; a = b; b = temp; }\n",
    "            { Py_ssize_t size_temp = size_a;\n",
    "                size_a = size_b;\n",
    "                size_b = size_temp; }\n",
    "        }\n",
    "        z = _PyLong_New(size_a+1);\n",
    "        if (z == NULL)\n",
    "            return NULL;\n",
    "        for (i = 0; i < size_b; ++i) {\n",
    "            carry += a->ob_digit[i] + b->ob_digit[i];\n",
    "            z->ob_digit[i] = carry & PyLong_MASK;\n",
    "            carry >>= PyLong_SHIFT;\n",
    "        }\n",
    "        for (; i < size_a; ++i) {\n",
    "            carry += a->ob_digit[i];\n",
    "            z->ob_digit[i] = carry & PyLong_MASK;\n",
    "            carry >>= PyLong_SHIFT;\n",
    "        }\n",
    "        z->ob_digit[i] = carry;\n",
    "        return long_normalize(z);\n",
    "    }\n",
    "    \n",
    "先解释函数中的关键局部变量：\n",
    "\n",
    "- size_a 和 size_b ，分别是操作数 a 和 b 底层数组的长度；\n",
    "- z ，用于保存计算结果的新整数对象；\n",
    "- i ，用于遍历底层整数数组；\n",
    "- carry ，用于保存每个字部分运算的进位；\n",
    "\n",
    "x_add 计算逻辑与此一模一样：\n",
    "\n",
    "- 第 10-15 行，如果 a 数组长度比较小，将 a 、 b 交换，数组长度较大的那个在前面；\n",
    "- 第 16-18 行，创建新整数对象，用于保存计算结果（注意到长度必须比 a 和 b 都大一，因为可能有进位）；\n",
    "- 第 19-23 行，遍历 b 底层数组，与 a 对应部分相加并保存到 z 中，需要特别注意进位计算；\n",
    "- 第 24-28 行，遍历 a 底层数组剩余部分，与进位相加后保存到 z 中，同样需要特别注意进位计算；\n",
    "- 第 29 行，将进位写入 z 底层数组最高位单元中；\n",
    "- 第 30 行，去除计算结果 z 底层数组中前面多余的零，因为最后的进位（ 29 行处）可能为零；\n",
    "\n",
    "最后，我们以一个例子图解两个整数 a 和 b 绝对值之和的计算过程：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_10.png\" width=800 height=600>  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 绝对值减法`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "有了绝对值加法的基础，绝对值减法才更容易弄懂。 x_sub 同样在 Objects/longobject.c 中实现：\n",
    "\n",
    "    static PyLongObject *\n",
    "    x_sub(PyLongObject *a, PyLongObject *b)\n",
    "    {\n",
    "        Py_ssize_t size_a = Py_ABS(Py_SIZE(a)), size_b = Py_ABS(Py_SIZE(b));\n",
    "        PyLongObject *z;\n",
    "        Py_ssize_t i;\n",
    "        int sign = 1;\n",
    "        digit borrow = 0;\n",
    "\n",
    "        /* Ensure a is the larger of the two: */\n",
    "        if (size_a < size_b) {\n",
    "            sign = -1;\n",
    "            { PyLongObject *temp = a; a = b; b = temp; }\n",
    "            { Py_ssize_t size_temp = size_a;\n",
    "                size_a = size_b;\n",
    "                size_b = size_temp; }\n",
    "        }\n",
    "        else if (size_a == size_b) {\n",
    "            /* Find highest digit where a and b differ: */\n",
    "            i = size_a;\n",
    "            while (--i >= 0 && a->ob_digit[i] == b->ob_digit[i])\n",
    "                ;\n",
    "            if (i < 0)\n",
    "                return (PyLongObject *)PyLong_FromLong(0);\n",
    "            if (a->ob_digit[i] < b->ob_digit[i]) {\n",
    "                sign = -1;\n",
    "                { PyLongObject *temp = a; a = b; b = temp; }\n",
    "            }\n",
    "            size_a = size_b = i+1;\n",
    "        }\n",
    "        z = _PyLong_New(size_a);\n",
    "        if (z == NULL)\n",
    "            return NULL;\n",
    "        for (i = 0; i < size_b; ++i) {\n",
    "            /* The following assumes unsigned arithmetic\n",
    "               works module 2**N for some N>PyLong_SHIFT. */\n",
    "            borrow = a->ob_digit[i] - b->ob_digit[i] - borrow;\n",
    "            z->ob_digit[i] = borrow & PyLong_MASK;\n",
    "            borrow >>= PyLong_SHIFT;\n",
    "            borrow &= 1; /* Keep only one sign bit */\n",
    "        }\n",
    "        for (; i < size_a; ++i) {\n",
    "            borrow = a->ob_digit[i] - borrow;\n",
    "            z->ob_digit[i] = borrow & PyLong_MASK;\n",
    "            borrow >>= PyLong_SHIFT;\n",
    "            borrow &= 1; /* Keep only one sign bit */\n",
    "        }\n",
    "        assert(borrow == 0);\n",
    "        if (sign < 0) {\n",
    "            Py_SIZE(z) = -Py_SIZE(z);\n",
    "        }\n",
    "        return long_normalize(z);\n",
    "    }\n",
    "\n",
    "同样，我们先搞清楚 x_sub 函数关键局部变量的作用，现列举如下：\n",
    "\n",
    "- size_a 和 size_b ，分别是操作数 a 和 b 底层数组的长度；\n",
    "- z ，用于保存计算结果的新整数对象；\n",
    "- i ，用于遍历底层整数数组；\n",
    "- sign ，绝对值减法结果符号， 1 表示正； -1 表示负；\n",
    "- borrow ，用于维护每部分减法操作的 借位 ；\n",
    "\n",
    "x_sub 计算绝对值之差的步骤如下：\n",
    "\n",
    "1. 第 11-17 行，如果 a 底层数组比 b 小，将计算结果设为 负 ( sign 为 -1 )，并将两者交换以方便计算；\n",
    "2. 第 18-30 行，如果 a 、 b 底层数组长度一样：\n",
    "\n",
    "    a. 第 20-22 行，跳过高位相同的部分；\n",
    "    \n",
    "    b. 第 23-24 行，如果每个部分都一样，则结果为0；\n",
    "    \n",
    "    c. 第 25-28 行，如果剩余最高部分， a 比较小，将计算结果设为 负 ，同样将两者交换以方便计算；\n",
    "\n",
    "3. 第 34*-41* 行，遍历 b 底层数组，与 a 对应部分相减并保存到 z 中，需要特别注意借位计算；\n",
    "4. 第 42*-47* 行，遍历 a 底层数组剩余部分，与借位相减后保存到 z 中，同样需要特别注意借位计算；\n",
    "5. 第 49-51 行，为计算结果 z 设置符号；\n",
    "6. 第 52 行，调用 long_normalize 剔除底层数组高位部分多余的零；\n",
    "\n",
    "\n",
    "    static PyLongObject *\n",
    "    x_sub(PyLongObject *a, PyLongObject *b)\n",
    "    {\n",
    "        Py_ssize_t size_a = Py_ABS(Py_SIZE(a)), size_b = Py_ABS(Py_SIZE(b));\n",
    "        PyLongObject *z;\n",
    "        Py_ssize_t i;\n",
    "        int sign = 1;\n",
    "        digit borrow = 0;\n",
    "\n",
    "        /* Ensure a is the larger of the two: */\n",
    "        if (size_a < size_b) {\n",
    "            sign = -1;\n",
    "            { PyLongObject *temp = a; a = b; b = temp; }\n",
    "            { Py_ssize_t size_temp = size_a;\n",
    "                size_a = size_b;\n",
    "                size_b = size_temp; }\n",
    "        }\n",
    "        else if (size_a == size_b) {\n",
    "            /* Find highest digit where a and b differ: */\n",
    "            i = size_a;\n",
    "            while (--i >= 0 && a->ob_digit[i] == b->ob_digit[i])\n",
    "                ;\n",
    "            if (i < 0)\n",
    "                return (PyLongObject *)PyLong_FromLong(0);\n",
    "            if (a->ob_digit[i] < b->ob_digit[i]) {\n",
    "                sign = -1;\n",
    "                { PyLongObject *temp = a; a = b; b = temp; }\n",
    "            }\n",
    "            size_a = size_b = i+1;\n",
    "        }\n",
    "        z = _PyLong_New(size_a);\n",
    "        if (z == NULL)\n",
    "            return NULL;\n",
    "        for (i = 0; i < size_b; ++i) {\n",
    "            /* The following assumes unsigned arithmetic works module 2**N for some N>PyLong_SHIFT. */\n",
    "            borrow = a->ob_digit[i] - b->ob_digit[i] - borrow;\n",
    "            z->ob_digit[i] = borrow & PyLong_MASK;\n",
    "            borrow >>= PyLong_SHIFT;\n",
    "            borrow &= 1; /* Keep only one sign bit */\n",
    "        }\n",
    "        for (; i < size_a; ++i) {\n",
    "            borrow = a->ob_digit[i] - borrow;\n",
    "            z->ob_digit[i] = borrow & PyLong_MASK;\n",
    "            borrow >>= PyLong_SHIFT;\n",
    "            borrow &= 1; /* Keep only one sign bit */\n",
    "        }\n",
    "        assert(borrow == 0);\n",
    "        if (sign < 0) {\n",
    "            Py_SIZE(z) = -Py_SIZE(z);\n",
    "        }\n",
    "        return long_normalize(z);\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 减法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "现在我们回过头来研究减法处理函数 long_sub ，函数同样位于源码文件 Objects/longobject.c 中：\n",
    "    \n",
    "    static PyObject *\n",
    "    long_sub(PyLongObject *a, PyLongObject *b)\n",
    "    {\n",
    "        PyLongObject *z;\n",
    "\n",
    "        CHECK_BINOP(a, b);\n",
    "\n",
    "        if (Py_ABS(Py_SIZE(a)) <= 1 && Py_ABS(Py_SIZE(b)) <= 1) {\n",
    "            return PyLong_FromLong(MEDIUM_VALUE(a) - MEDIUM_VALUE(b));\n",
    "        }\n",
    "        if (Py_SIZE(a) < 0) {\n",
    "            if (Py_SIZE(b) < 0)\n",
    "                z = x_sub(a, b);\n",
    "            else\n",
    "                z = x_add(a, b);\n",
    "            if (z != NULL) {\n",
    "                assert(Py_SIZE(z) == 0 || Py_REFCNT(z) == 1);\n",
    "                Py_SIZE(z) = -(Py_SIZE(z));\n",
    "            }\n",
    "        }\n",
    "        else {\n",
    "            if (Py_SIZE(b) < 0)\n",
    "                z = x_add(a, b);\n",
    "            else\n",
    "                z = x_sub(a, b);\n",
    "        }\n",
    "        return (PyObject *)z;\n",
    "    }\n",
    "    \n",
    "long_sub 跟 long_add 一样直观，基于绝对值加法 x_add 和绝对值减法 x_sub 这两个底层操作实现：\n",
    "\n",
    "1. 第 8-10 行，如果 a 、 b 底层数组长度均不超过 1 ，直接转换成 C 基本整数类型进行计算；\n",
    "2. 第 13 行，如果 a 、 b 均为负数，结果为 |b|-|a|∣b∣−∣a∣ ，即-(|a|-|b|)−(∣a∣−∣b∣) ，特别留意 18 行处；\n",
    "3. 第 15 行，如果 a 为负数， b 为正数，结果为 -(|a|+|b|)−(∣a∣+∣b∣) ，特别留意 18 行处；\n",
    "4. 第 23 行，如果 a 为正数， b 为负数，结果为 |a|+|b|∣a∣+∣b∣；\n",
    "5. 第 25 行，如果 a 、 b 均为正数，结果为 |a|-|b|∣a∣−∣b∣ ；\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_11.png\" width=500 height=400>  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## bytes 对象，不可变的字节序列"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "不少编程语言中的 字符串 都是由 字符数组 (或称为 字节序列 )来表示， C 语言就是这样。\n",
    "\n",
    "    char msg[] = \"Hello, world!\";\n",
    "    \n",
    "由于一个字节最多只能表示 256 种字符，用来表示英文字符绰绰有余，想覆盖非英文字符便捉襟见肘了。为了表示众多的非英文字符(比如汉字)，计算机先驱们发明了**多字节编码** ——通过多个字节来表示一个字符。由于原始字节序列不维护编码信息，操作不慎便导致各种乱码现象。\n",
    "\n",
    "Python 提供的解决方案是 Unicode 字符串 ( str )对象， Unicode 可以表示各种字符，无需关心编码。然而存储或者网络通讯时，字符串对象不可避免要**序列化**成字节序列。为此， Python 额外提供了字节序列对象—— bytes 。\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_12.png\" width=800 height=600>  \n",
    "\n",
    "如上图， str 对象统一表示一个**字符串**，不需要关心编码；计算机通过**字节序列**与存储介质和网络介质打交道，字节序列由 bytes 对象表示；存储或传输 str 对象时，需要将其**序列化**成字节序列，序列化过程也是*8编码**的过程。\n",
    "\n",
    "好了，我们已经弄明白 str 对象与 bytes 之间的关系，这两者是 Python 中最重要的内建对象之一。 读者对 str 对象应该再熟悉不过了，但对更接近底层的 bytes 对象可能涉猎不多。没关系，经过本节学习，你将彻底掌握它！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 对象结构"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "bytes 对象用于表示由若干字节组成的**字节序列**以及相关的**操作**，并不关心字节序列的**含义**。因此， bytes 应该是一种**变长对象**，内部由C数组实现。 Include/bytesobject.h 头文件中的定义印证了我们的猜测：\n",
    "\n",
    "    typedef struct {\n",
    "        PyObject_VAR_HEAD\n",
    "        Py_hash_t ob_shash;\n",
    "        char ob_sval[1];\n",
    "\n",
    "        /* Invariants:\n",
    "         *     ob_sval contains space for 'ob_size+1' elements.\n",
    "         *     ob_sval[ob_size] == 0.\n",
    "         *     ob_shash is the hash of the string or -1 if not computed yet.\n",
    "         */\n",
    "    } PyBytesObject;\n",
    "    \n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_13.png\" width=800 height=600>  \n",
    "\n",
    "字节序列对象 PyBytesObject 中，确实藏着一个字符数组ob_sval。注意到 ob_sval 数组长度定义为 1 ，这是 C 语言中定义**变长数组**的技巧。这个技巧在前面章节( int 对象，永不溢出的整数 )中介绍过，这里不再赘述。源码注释表明， Python 为待存储的字节序列额外分配一个字节，用于在末尾处保存 \\0 ，以便兼容 C 字符串。\n",
    "\n",
    "此外，我们还留意到另一个字段 ob_shash ，它用于保存字节序列的**哈希值**。 Python 对象哈希值应用范围很广，比如 dict 字典对象依赖对象哈希值进行存储。由于计算 bytes 对象哈希值需要遍历其内部的字符数组，开销相对较大。因此， Python 选择将哈希值保存起来，以空间换时间，避免重复计算。\n",
    "\n",
    "最后，以几个典型例子结束 bytes 对象结构介绍，以此加深理解：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_14.png\" width=800 height=600>  \n",
    "\n",
    "由此可见，就算空 bytes 对象( b'' )也是要占用内存空间的，至少变长对象 公共头部 是少不了的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-25T08:50:46.178193Z",
     "start_time": "2021-03-25T08:50:46.166200Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "sys.getsizeof(b'')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "bytes 对象占用的内存空间可分为以下个部分进行计算：\n",
    "\n",
    "- 变长对象公共头部 24 字节，ob_refcnt 、 ob_type 、 ob_size 每个字段各占用 8 字节；\n",
    "- 哈希值 ob_shash 占用 8 字节；\n",
    "- 字节序列本身，假设是 n 字节；\n",
    "- 额外 1 字节用于存储末尾处的 \\0 ；\n",
    "\n",
    "因此， bytes 对象空间计算公式为 24+8+n+124+8+n+1，即 33+n33+n，其中 n 为字节序列长度。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 对象行为"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "现在，我们开始考察 bytes 对象的 行为 。由于对象的行为由对象的 类型 决定，因而我们需要到 bytes 类型对象中寻找答案。在 Objects/bytesobject.c 源码文件中，我们找到 bytes 类型对象 的定义：\n",
    "\n",
    "    PyTypeObject PyBytes_Type = {\n",
    "        PyVarObject_HEAD_INIT(&PyType_Type, 0)\n",
    "        \"bytes\",\n",
    "        PyBytesObject_SIZE,\n",
    "        sizeof(char),\n",
    "        // ...\n",
    "        &bytes_as_number,                           /* tp_as_number */\n",
    "        &bytes_as_sequence,                         /* tp_as_sequence */\n",
    "        &bytes_as_mapping,                          /* tp_as_mapping */\n",
    "        (hashfunc)bytes_hash,                       /* tp_hash */\n",
    "        // ...\n",
    "    };\n",
    "    \n",
    "我们对类型对象的内部结构已经非常熟悉了， tp_as_xxxx 系列结构体决定了对象支持的各种 操作 。举个例子， bytes_as_number 结构体中保存着**数值运算**处理函数的指针。 bytes 对象居然支持数据操作，没搞错吧？我们看到， bytes_as_number 结构体中只定义了一个操作—— 模运算 ( % )：\n",
    "\n",
    "    static PyNumberMethods bytes_as_number = {\n",
    "        0,              /*nb_add*/\n",
    "        0,              /*nb_subtract*/\n",
    "        0,              /*nb_multiply*/\n",
    "        bytes_mod,      /*nb_remainder*/\n",
    "    }\n",
    "    \n",
    "    static PyObject *\n",
    "    bytes_mod(PyObject *self, PyObject *arg)\n",
    "    {\n",
    "        if (!PyBytes_Check(self)) {\n",
    "            Py_RETURN_NOTIMPLEMENTED;\n",
    "        }\n",
    "        return _PyBytes_FormatEx(PyBytes_AS_STRING(self), PyBytes_GET_SIZE(self),\n",
    "                                 arg, 0);\n",
    "    }\n",
    "    \n",
    "由此可见， bytes 对象只是借用 % 运算符实现字符串格式化，谈不上支持数值运算，虚惊一场：\n",
    "\n",
    "    >>> b'msg: a=%d b=%d' % (1, 2)\n",
    "    b'msg: a=1 b=2'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 序列型操作"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "众所周知， bytes 是 序列型对象 ，序列型操作才是研究重点。我们在 bytes_as_sequence 结构体中找到相关定义：\n",
    "\n",
    "    static PySequenceMethods bytes_as_sequence = {\n",
    "    (lenfunc)bytes_length, /*sq_length*/\n",
    "    (binaryfunc)bytes_concat, /*sq_concat*/\n",
    "    (ssizeargfunc)bytes_repeat, /*sq_repeat*/\n",
    "    (ssizeargfunc)bytes_item, /*sq_item*/\n",
    "    0,                  /*sq_slice*/\n",
    "    0,                  /*sq_ass_item*/\n",
    "    0,                  /*sq_ass_slice*/\n",
    "    (objobjproc)bytes_contains /*sq_contains*/\n",
    "    };\n",
    "    \n",
    "由此可见， bytes 支持的 序列型操作 包括以下 5 个：\n",
    "\n",
    "- sq_length ，查询序列长度；\n",
    "- sq_concat ，将两个序列合并为一个；\n",
    "- sq_repeat ，将序列重复多次；\n",
    "- sq_item ，取出给定下标序列元素；\n",
    "- sq_contains，包含关系判断；"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### 长度"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "最简单的序列型操作是 长度查询 ，直接返回 ob_size 字段即可：\n",
    "\n",
    "    static Py_ssize_t\n",
    "    bytes_length(PyBytesObject *a)\n",
    "    {\n",
    "        return Py_SIZE(a);\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### 合并"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-25T08:57:40.366189Z",
     "start_time": "2021-03-25T08:57:40.352197Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "print(b'abc' + b'cba')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "合并操作将两个 bytes 对象拼接成一个，由 bytes_concat 函数处理：\n",
    "\n",
    "    static PyObject *\n",
    "    bytes_concat(PyObject *a, PyObject *b)\n",
    "    {\n",
    "        Py_buffer va, vb;\n",
    "        PyObject *result = NULL;\n",
    "\n",
    "        va.len = -1;\n",
    "        vb.len = -1;\n",
    "        if (PyObject_GetBuffer(a, &va, PyBUF_SIMPLE) != 0 ||\n",
    "            PyObject_GetBuffer(b, &vb, PyBUF_SIMPLE) != 0) {\n",
    "            PyErr_Format(PyExc_TypeError, \"can't concat %.100s to %.100s\",\n",
    "                         Py_TYPE(b)->tp_name, Py_TYPE(a)->tp_name);\n",
    "            goto done;\n",
    "        }\n",
    "\n",
    "        /* Optimize end cases */\n",
    "        if (va.len == 0 && PyBytes_CheckExact(b)) {\n",
    "            result = b;\n",
    "            Py_INCREF(result);\n",
    "            goto done;\n",
    "        }\n",
    "        if (vb.len == 0 && PyBytes_CheckExact(a)) {\n",
    "            result = a;\n",
    "            Py_INCREF(result);\n",
    "            goto done;\n",
    "        }\n",
    "\n",
    "        if (va.len > PY_SSIZE_T_MAX - vb.len) {\n",
    "            PyErr_NoMemory();\n",
    "            goto done;\n",
    "        }\n",
    "\n",
    "        result = PyBytes_FromStringAndSize(NULL, va.len + vb.len);\n",
    "        if (result != NULL) {\n",
    "            memcpy(PyBytes_AS_STRING(result), va.buf, va.len);\n",
    "            memcpy(PyBytes_AS_STRING(result) + va.len, vb.buf, vb.len);\n",
    "        }\n",
    "\n",
    "      done:\n",
    "        if (va.len != -1)\n",
    "            PyBuffer_Release(&va);\n",
    "        if (vb.len != -1)\n",
    "            PyBuffer_Release(&vb);\n",
    "        return result;\n",
    "    }\n",
    "    \n",
    "1. 第 4-5 行，定义局部变量 va 、 vb 用于维护缓冲区， result 用于保存合并结果；\n",
    "2. 第 7-14 行，从待合并对象中获取字节序列所在缓冲区；\n",
    "3. 第 17-21 行，如果第一个对象长度为 0 ，第二个对象就是结果；\n",
    "4. 第 22-26 行，反之第二个对象长度为 0 ，第一个对象就是结果；\n",
    "5. 第 28-31 行，长度超过限制则报错，其实判断条件这样写更直观：va.len + vb.len > PY_SSIZE_T_MAX ；\n",
    "6. 第 33 行，新建 bytes 对象用于保存合并结果，长度为待合并对象长度之和；\n",
    "7. 第 34-37 行，将字节序列从待合并对象拷贝到结果对象；\n",
    "8. 第 39-44 行，返回结果。\n",
    "\n",
    "Py_buffer 提供了一套操作对象缓冲区的统一接口，屏蔽不同类型对象的内部差异：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_15.png\" width=800 height=600>  \n",
    "bytes_concat 函数逻辑很直白，将两个 bytes 对象的缓冲区拷贝到一起形成新 bytes 对象。 sq_repeat 等其他处理函数也不复杂，因篇幅关系不再单独讲解了。鼓励读者们自行深入源码，弄清他们的来龙去脉，必有收获。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### 数据拷贝的陷阱"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "bytes 对象提供了一个内建方法 join ，可高效合并多个 bytes 对象：\n",
    "    \n",
    "    result = b''.join(segments)\n",
    "    \n",
    "join 方法对数据拷贝进行了优化：先遍历待合并对象，计算总长度；然后根据总长度创建目标对象；最后再遍历待合并对象，逐一拷贝数据。这样一来，每个对象均只需拷贝一次，解决了重复拷贝的陷阱\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_16.png\" width=800 height=600>  \n",
    "join 内建方法同样在 Objects/bytesobject.c 文件中实现，bytes_join 是也，这里不再展开介绍了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 字符缓冲池"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "为了优化单字节 bytes 对象(也可称为 字符对象 )的创建效率， Python 内部维护了一个 字符缓冲池 ：\n",
    "\n",
    "    static PyBytesObject *characters[UCHAR_MAX + 1];\n",
    "\n",
    "Python 内部创建单字节 bytes 对象时，先检查目标对象是否已在缓冲池中。PyBytes_FromStringAndSize 函数是负责创建 bytes 对象的通用接口，同样位于 Objects/bytesobject.c 中：\n",
    "\n",
    "    PyObject *\n",
    "    PyBytes_FromStringAndSize(const char *str, Py_ssize_t size)\n",
    "    {\n",
    "        PyBytesObject *op;\n",
    "        if (size < 0) {\n",
    "            PyErr_SetString(PyExc_SystemError,\n",
    "                \"Negative size passed to PyBytes_FromStringAndSize\");\n",
    "            return NULL;\n",
    "        }\n",
    "        if (size == 1 && str != NULL &&\n",
    "            (op = characters[*str & UCHAR_MAX]) != NULL)\n",
    "        {\n",
    "    #ifdef COUNT_ALLOCS\n",
    "            one_strings++;\n",
    "    #endif\n",
    "            Py_INCREF(op);\n",
    "            return (PyObject *)op;\n",
    "        }\n",
    "\n",
    "        op = (PyBytesObject *)_PyBytes_FromSize(size, 0);\n",
    "        if (op == NULL)\n",
    "            return NULL;\n",
    "        if (str == NULL)\n",
    "            return (PyObject *) op;\n",
    "\n",
    "        memcpy(op->ob_sval, str, size);\n",
    "        /* share short strings */\n",
    "        if (size == 1) {\n",
    "            characters[*str & UCHAR_MAX] = op;\n",
    "            Py_INCREF(op);\n",
    "        }\n",
    "        return (PyObject *) op;\n",
    "    }\n",
    "    \n",
    "其中，涉及字符缓冲区维护的关键步骤是：\n",
    "\n",
    "- 第 10-18 行，如果目标对象为单字节对象且已在字符缓冲池中，直接返回已缓存对象；\n",
    "- 第 20-26 行，创建新 bytes 对象并拷贝字节序列；\n",
    "- 第 28-31 行，如果创建的对象为单字节对象，将其放入字符缓冲池；\n",
    "\n",
    "由此可见，当 Python 程序开始运行时，字符缓冲池是空的。随着单字节 bytes 对象的创建，缓冲池中的对象慢慢多了起来。当缓冲池已缓存 b'1' 、b'2' 、b'3' 、b'a' 、b'b' 、b'c' 这几个字符时，内部结构如下：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_17.png\" width=800 height=600>  \n",
    "这样一来，字符对象首次创建后便在缓冲池中缓存起来；后续再次使用时， Python 直接从缓冲池中取，避免重复创建和销毁。与前面章节介绍的**小整数**一样，字符对象只有为数不多的**256**个，但使用频率非常高。缓冲池技术作为一种**空间换时间**的优化手段，只需较小的内存为代价，便可明显提升执行效率。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 总结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Python 中的字符串主要由 str 和 bytes 这两个内建对象来承载，本节我们一起研究了更靠近底层的 bytes 对象：\n",
    "\n",
    "- bytes 是一种 **变长** 、 **不可变** 对象，内部由一个 C 字符数组实现；\n",
    "- bytes 也是 序列型对象 ，它支持的 序列操作 在 bytes_as_sequence 中定义；\n",
    "- Python 内部维护**字符缓冲池**优化单字节 bytes 对象的创建和销毁操作；\n",
    "- **缓冲池**是一种常用的**以空间换时间**的优化技术 ；"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## str 对象，统一的 Unicode 字符串"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Unicode是什么"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "计算机存储的基本单位是 八位字节 ，由 8 个比特位组成，简称 字节 。由于英文只由 26 个字母加若干符号组成，因此英文字符可以直接用 字节 来保存。其他诸如中日韩等语言，由于字符众多，则不得不用多个字节来编码。\n",
    "\n",
    "随着计算机技术的传播，非拉丁文字符编码技术蓬勃发展，但存在两个比较大的局限性：\n",
    "\n",
    "不支持多语言 ，例如中文的编码方案不能表示日文；\n",
    "没有统一标准 ，例如中文有 GB2312 ，GBK 、 GB18030 等多种编码标准；\n",
    "由于编码方式不统一，开发人员经常需要在不同编码间来回转化，错误频出。为了彻底解决这些问题， 统一码联盟 提出了 Unicode 标准。Unicode 对世界上大部分文字系统进行整理、编码，让计算机可以用统一的方式处理文本。Unicode 目前已经收录了超过 13 万个字符，天然地支持多语言。使用 Unicode ，即可彻底跟编码问题说拜拜！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Python中的Unicode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Python 在 3 之后，str 对象内部改用 Unicode 表示，因而被源码称为 Unicode 对象。这么做好处是显然易见的，程序核心逻辑统一用 Unicode ，只需在输入、输入层进行编码、解码，可最大程度避免各种编码问题：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_18.jpg\" width=800 height=600>  \n",
    "由于 Unicode 收录字符已经超过 13 万个，每个字符至少需要 4 个字节来保存。这意味着巨大的内存开销，显然是不可接受的。英文字符用 ASCII 表示仅需 1 个字节，而用 Unicode 表示内存开销却增加 4 倍！\n",
    "\n",
    "Python 作者们肯定不允许这样的事情发生，不信我们先来观察下( getsizeof 获取对象内存大小)："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### PyASCIIObject"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "如果 str 对象保存的文本均为 ASCII ，即 maxchar<128，则底层由 PyASCIIObject 结构存储：\n",
    "\n",
    "    /* ASCII-only strings created through PyUnicode_New use the PyASCIIObject\n",
    "       structure. state.ascii and state.compact are set, and the data\n",
    "       immediately follow the structure. utf8_length and wstr_length can be found\n",
    "       in the length field; the utf8 pointer is equal to the data pointer. */\n",
    "    typedef struct {\n",
    "        PyObject_HEAD\n",
    "        Py_ssize_t length;          /* Number of code points in the string */\n",
    "        Py_hash_t hash;             /* Hash value; -1 if not set */\n",
    "        struct {\n",
    "            unsigned int interned:2;\n",
    "            unsigned int kind:3;\n",
    "            unsigned int compact:1;\n",
    "            unsigned int ascii:1;\n",
    "            unsigned int ready:1;\n",
    "            unsigned int :24;\n",
    "        } state;\n",
    "        wchar_t *wstr;              /* wchar_t representation (null-terminated) */\n",
    "    } PyASCIIObject;\n",
    "\n",
    "\n",
    "PyASCIIObject 结构体也是其他 Unicode 底层存储结构体的基础，所有字段均为 Unicode 公共字段：\n",
    "\n",
    "- ob_refcnt ，引用计数；\n",
    "- ob_type ，对象类型；\n",
    "- length ，文本长度；\n",
    "- hash ，文本哈希值；\n",
    "- state ，Unicode 对象标志位，包括 internel 、 kind 、 ascii 、 compact 等；\n",
    "- wstr ，略；\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_19.jpg\" width=800 height=600>  \n",
    "\n",
    "注意到，state 字段后有一个 4 字节的空洞，这是结构体字段 内存对齐 造成的现象。在 64 位机器下，指针大小为 8 字节，为优化内存访问效率，wstr 必须以 8 字节对齐；而 state 字段大小只是 4 字节，便留下 4 字节的空洞。PyASCIIObject 结构体大小在 64 位机器下为 48 字节，在 32 位机器下为 24 字节。\n",
    "\n",
    "ASCII 文本则紧接着位于 PyASCIIObject 结构体后面，以字符串对象 ‘abc’ 以及空字符串对象 ‘’ 为例：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_20.jpg\" width=800 height=600>  \n",
    "\n",
    "注意到，与 bytes 对象一样，Python 也在 ASCII 文本末尾，额外添加一个 \\0 字符，以兼容 C 字符串。\n",
    "\n",
    "如此一来，以 Unicode 表示的 ASCII 文本，额外内存开销仅为 PyASCIIObject 结构体加上末尾的 \\0 字节而已。PyASCIIObject 结构体在 64 位机器下，大小为 48 字节。因此，长度为 n 的纯 ASCII 字符串对象，需要消耗 n+48+1，即 n+49 字节的内存空间。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### PyCompactUnicodeObject"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "如果文本不全是 ASCII ，Unicode 对象底层便由 PyCompactUnicodeObject 结构体保存：\n",
    "\n",
    "    /* Non-ASCII strings allocated through PyUnicode_New use the\n",
    "       PyCompactUnicodeObject structure. state.compact is set, and the data\n",
    "       immediately follow the structure. */\n",
    "    typedef struct {\n",
    "        PyASCIIObject _base;\n",
    "        Py_ssize_t utf8_length;     /* Number of bytes in utf8, excluding the\n",
    "                                     * terminating \\0. */\n",
    "        char *utf8;                 /* UTF-8 representation (null-terminated) */\n",
    "        Py_ssize_t wstr_length;     /* Number of code points in wstr, possible\n",
    "                                     * surrogates count as two code points. */\n",
    "    } PyCompactUnicodeObject;\n",
    "    \n",
    "PyCompactUnicodeObject 在 PyASCIIObject 基础上，增加 3 个字段：\n",
    "\n",
    "- utf8_length ，文本 UTF8 编码长度；\n",
    "- utf8 ，文本 UTF8 编码形式，缓存以避免重复编码运算；\n",
    "- wstr_length ，略；\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_21.jpg\" width=800 height=600>  \n",
    "\n",
    "由于 ASCII 本身是合法的 UTF8 ，无须保存 UTF8 编码形式，这也是 ASCII 文本底层由 PyASCIIObject 保存的原因。在 64 位机器，PyCompactUnicodeObject 结构体大小为 72 字节；在 32 位机器则是 36 字节。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### PyUnicode_1BYTE_KIND"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "如果 128<=maxchar<=<256，Unicode 对象底层便由 PyCompactUnicodeObject 结构体保存，字符存储单元为 Py_UCS1 ，大小为 1 字节。以 Python® 为例，字符 ® 码位为 U+00AE ，满足该条件，内部结构如下：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_23.jpg\" width=800 height=600>  \n",
    "\n",
    "由于 ASCII 本身是合法的 UTF8 ，无须保存 UTF8 编码形式，这也是 ASCII 文本底层由 PyASCIIObject 保存的原因。在 64 位机器，PyCompactUnicodeObject 结构体大小为 72 字节；在 32 位机器则是 36 字节。\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_22.jpg\" width=800 height=600>  \n",
    "\n",
    "字符存储单元还是 1 字节，跟 ASCII 文本一样。 因此，Python® 对象需要占用 80 字节的内存空间72+1*7+1=72+8=8072+1∗7+1=72+8=80："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-28T08:03:42.082366Z",
     "start_time": "2021-03-28T08:03:42.063368Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.getsizeof('Python®')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### PyUnicode_2BYTE_KIND"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "如果 256<=maxchar<=65536，Unicode 对象底层同样由 PyCompactUnicodeObject 结构体保存，但字符存储单元为Py_UCS2，大小为 2 字节。以 AC米兰 为例，常用汉字码位在 U+0100 到 U+FFFF 之间，满足该条件，内部结构如下：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_24.jpg\" width=800 height=600>  \n",
    "由于现在字符存储单元为 2 字节，故而 str 对象 AC米兰 需要占用 82 字节的内存空间：72+2*4+2=72+10=8272+2∗4+2=72+10=82maxchar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-28T08:04:06.161924Z",
     "start_time": "2021-03-28T08:04:06.143925Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "sys.getsizeof('AC米兰')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "我们看到，当文本包含中文后，英文字母也只能用 2 字节的存储单元来保存了。\n",
    "\n",
    "你可能会提出疑问，为什么不采用变长存储单元呢？例如，字母 1 字节，汉字 2 字节？这是因为采用变长存储单元后，就无法在 O(1) 时间内取出文本第n个字符了——你只能从头遍历直到遇到第 n 个字符。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### PyUnicode_4BYTE_KIND"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "如果 $65536<=maxchar<429496296$，便只能用 4 字节存储单元 Py_UCS4 了。以'AC米兰emoji.emojize(':thumbs_up:')'为例：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_25.jpg\" width=800 height=600>  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-28T08:04:30.057824Z",
     "start_time": "2021-03-28T08:04:30.032827Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "import emoji\n",
    "sys.getsizeof('AC米兰'+emoji.emojize(':thumbs_up:'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "这样一来，给一段英文文本加上表情，内存暴增 4 倍，也就不奇怪了："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-28T08:04:43.145940Z",
     "start_time": "2021-03-28T08:04:43.135941Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "text = 'a' * 1000\n",
    "sys.getsizeof(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-28T08:04:43.900450Z",
     "start_time": "2021-03-28T08:04:43.885452Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "text += emoji.emojize(':thumbs_up:')\n",
    "sys.getsizeof(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### interned机制"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "如果 str 对象 interned 标识位为 1 ，Python 虚拟机将为其开启 interned 机制。那么，什么是 interned 机制？\n",
    "\n",
    "先考虑以下场景，如果程序中有大量 User 对象，有什么可优化的地方？\n",
    "\n",
    "    >>> class User:\n",
    "    ...\n",
    "    ...     def __init__(self, name, age):\n",
    "    ...         self.name = name\n",
    "    ...         self.age = age\n",
    "    ...\n",
    "    >>>\n",
    "    >>> user = User(name='tom', age=20)\n",
    "    >>> user.__dict__\n",
    "    {'name': 'tom', 'age': 20}\n",
    "    \n",
    "由于对象的属性由 dict 保存，这意味着每个 User 对象都需要保存 str 对象 name 。换句话讲，1 亿个 User 对象需要重复保存 1 亿个同样的 str 对象，这将浪费多少内存！\n",
    "\n",
    "由于 str 是不可变对象，因此 Python 内部将有潜在重复可能的字符串都做成 单例模式 ，这就是 interned 机制。Python 具体做法是在内部维护一个全局 dict 对象，所有开启 interned 机制 str 对象均保存在这里；后续需要用到相关对象的地方，则优先到全局 dict 中取，避免重复创建。\n",
    "\n",
    "举个例子，虽然 str 对象 ‘abc’ 由不同的运算产生，但背后却是同一个对象："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-28T08:05:02.806434Z",
     "start_time": "2021-03-28T08:05:02.791435Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "a = 'abc'\n",
    "b = 'ab' + 'c'\n",
    "id(a), id(b), a is b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## list 对象，容量自适应的数组式容器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 基本用法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-28T08:05:35.203660Z",
     "start_time": "2021-03-28T08:05:35.184660Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# 新建一个列表\n",
    "l = [1, 2, 3]\n",
    "l\n",
    "[1, 2, 3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-28T08:05:38.617754Z",
     "start_time": "2021-03-28T08:05:38.607756Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# 向尾部追加元素，列表对象视情况自动扩容\n",
    ">>> l.append(4)\n",
    ">>> l\n",
    "[1, 2, 3, 4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-28T08:05:41.726936Z",
     "start_time": "2021-03-28T08:05:41.712937Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# 从尾部弹出元素，列表对象视情况自动缩容\n",
    ">>> l.pop()\n",
    "4\n",
    ">>> l\n",
    "[1, 2, 3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-28T08:05:44.972547Z",
     "start_time": "2021-03-28T08:05:44.960549Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# 向头部插入元素，该操作需要挪动后面的元素，谨慎使用！\n",
    ">>> l.insert(0, 4)\n",
    ">>> l\n",
    "[4, 1, 2, 3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-28T08:05:48.501820Z",
     "start_time": "2021-03-28T08:05:48.495824Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# 从头部弹出元素，该操作需要挪动后面的元素，谨慎使用！\n",
    ">>> l.pop(0)\n",
    "4\n",
    ">>> l\n",
    "[1, 2, 3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-28T08:05:53.174245Z",
     "start_time": "2021-03-28T08:05:53.168246Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# 查找元素第一次出现位置的下标\n",
    ">>> l.index(2)\n",
    "1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-28T08:05:56.946823Z",
     "start_time": "2021-03-28T08:05:56.927824Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# 用一个可迭代对象扩展列表——元素逐一追加到尾部\n",
    ">>> l.extend([1, 2])\n",
    ">>> l\n",
    "[1, 2, 3, 1, 2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-28T08:06:00.019714Z",
     "start_time": "2021-03-28T08:06:00.007715Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# 计算元素出现的个数\n",
    ">>> l.count(1)\n",
    "2\n",
    ">>> l.count(3)\n",
    "1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-28T08:06:03.313371Z",
     "start_time": "2021-03-28T08:06:03.296372Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# 将列表反转\n",
    ">>> l.reverse()\n",
    ">>> l\n",
    "[2, 1, 3, 2, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-28T08:06:08.065547Z",
     "start_time": "2021-03-28T08:06:08.056548Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# 将列表清空\n",
    ">>> l.clear()\n",
    ">>> l\n",
    "[]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "一个合格的 Python 开发工程师，除了必须熟练掌握 list 对象的基本操作，还需要对每个操作的 实现原理 及对应的 时间复杂度 、 空间复杂度 有准确的认识。列表操作总体比较简单，但有个操作特别容易被误用：\n",
    "\n",
    "- insert 方法向头部追加元素时需要挪动整个列表，时间复杂度是$O(n)$，性能极差，需谨慎使用；\n",
    "- append 方法向尾部追加元素时，无需挪动任何元素，时间复杂度$O(1)$；\n",
    "- pop 方法从头部弹出元素时也需要挪动整个列表，时间复杂度是$O(n)$，同样需谨慎使用；\n",
    "- pop 方法从尾部弹出元素时，无需挪动任何元素，时间复杂度是$O(1)$\n",
    "\n",
    "由此可见，对列表头部和尾部进行操作，性能有天壤之别。后续我们将一起探索 list 对象内部结构，从中寻找造成这种现象的原因。此外， list 对象还可根据元素个数 自动扩缩容 ，其中秘密也将一一揭晓。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 内部结构"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "list 对象在 Python 内部，由 PyListObject 结构体表示，定义于头文件 Include/listobject.h 中：\n",
    "\n",
    "    typedef struct {\n",
    "        PyObject_VAR_HEAD\n",
    "        /* Vector of pointers to list elements.  list[0] is ob_item[0], etc. */\n",
    "        PyObject **ob_item;\n",
    "\n",
    "        /* ob_item contains space for 'allocated' elements.  The number\n",
    "         * currently in use is ob_size.\n",
    "         * Invariants:\n",
    "         *     0 <= ob_size <= allocated\n",
    "         *     len(list) == ob_size\n",
    "         *     ob_item == NULL implies ob_size == allocated == 0\n",
    "         * list.sort() temporarily sets allocated to -1 to detect mutations.\n",
    "         *\n",
    "         * Items must normally not be NULL, except during construction when\n",
    "         * the list is not yet visible outside the function that builds it.\n",
    "         */\n",
    "        Py_ssize_t allocated;\n",
    "    } PyListObject;\n",
    "    \n",
    "毫无疑问， list 对象是一种 变长对象 ，因此包含变长对象公共头部。除了公共头部， list 内部维护了一个动态数组，而数组则依次保存元素对象的指针：\n",
    "\n",
    "- ob_item ，指向动态数组的指针，动态数组保存元素对象的指针；\n",
    "- allocated ，动态数组长度，即列表 容量 ；\n",
    "- ob_size ，动态数组当前保存元素个数，即列表 长度 。\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_26.jpg\" width=800 height=600>  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 尾部操作"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "如果 list 对象内部数组已用满，再添加元素时则需要进行扩容。 append 等方法在操作时都会对内部数组进行检查，如需扩容则调用 list_resize 函数。在 list_resize 函数， Python 重新分配一个长度更大的数组并替换旧数组。为避免频繁扩容， Python 每次都会为内部数组预留一定的裕量。\n",
    "\n",
    "假设列表元素l保存3个元素，内部数组长度为3，已满。当我们调用 append 方法向列表尾部追加元素时，需要对内部数组进行扩容。扩容步骤如下：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_28.jpg\" width=800 height=600>  \n",
    "\n",
    "1. 分配一个更大的数组，假设长度为 6 ，预留一定裕量避免频繁扩容；\n",
    "2. 将列表元素从旧数组逐一转移到新数组；\n",
    "3. 以新数组替换旧数组，并更新 allocated 字段；\n",
    "4. 回收旧数组。\n",
    "\n",
    "由于内部数组扩容时，需要将列表元素从旧数组拷贝到新数组，时间复杂度为$O(n)$ ，开销较大，需要尽量避免。为此， Python 在为内部数组扩容时，会预留一定裕量，一般是 1/8 左右。假设为长度为 1000 的列表对象扩容， Python 会预留大约 125 个空闲位置，分配一个长度 1125 的新数组。\n",
    "\n",
    "由于扩容操作的存在， append 方法最坏情况下时间复杂度为$O(n)$。由于扩容操作不会频繁发生，将扩容操作时的元素拷贝开销平摊到多个 append 操作中，平均时间复杂度还是$O(1)$。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### 动态扩容"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "如果 list 对象内部数组已用满，再添加元素时则需要进行扩容。 append 等方法在操作时都会对内部数组进行检查，如需扩容则调用 list_resize 函数。在 list_resize 函数， Python 重新分配一个长度更大的数组并替换旧数组。为避免频繁扩容， Python 每次都会为内部数组预留一定的裕量。\n",
    "\n",
    "假设列表元素l保存3个元素，内部数组长度为3，已满。当我们调用 append 方法向列表尾部追加元素时，需要对内部数组进行扩容。扩容步骤如下：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_28.jpg\" width=800 height=600>  \n",
    "\n",
    "1. 分配一个更大的数组，假设长度为 6 ，预留一定裕量避免频繁扩容；\n",
    "2. 将列表元素从旧数组逐一转移到新数组；\n",
    "3. 以新数组替换旧数组，并更新 allocated 字段；\n",
    "4. 回收旧数组。\n",
    "\n",
    "由于内部数组扩容时，需要将列表元素从旧数组拷贝到新数组，时间复杂度为$O(n)$ ，开销较大，需要尽量避免。为此， Python 在为内部数组扩容时，会预留一定裕量，一般是 1/8 左右。假设为长度为 1000 的列表对象扩容， Python 会预留大约 125 个空闲位置，分配一个长度 1125 的新数组。\n",
    "\n",
    "由于扩容操作的存在， append 方法最坏情况下时间复杂度为$O(n)$。由于扩容操作不会频繁发生，将扩容操作时的元素拷贝开销平摊到多个 append 操作中，平均时间复杂度还是$O(1)$。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 头部操作"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "与尾部相比，由于在列表头部增删元素需要挪动其他列表元素，性能有天地之别。\n",
    "\n",
    "假设列表元素 l 内部数组长度为 5 ，以及保存 3 个元素，分别是： 1 、 2 、 3 。当我们调用 insert 方法向头部插入元素时，需要先将当前所有元素往后挪一位，以便为新元素腾出一个空闲位置：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_29.jpg\" width=800 height=600>  \n",
    "\n",
    "然后， insert 方法将新元素存入挪出来的空闲位置，更新新 ob_size 字段，便完成了插入操作：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_30.jpg\" width=800 height=600>  \n",
    "\n",
    "因此， insert 在头部插入元素的时间复杂度是$O(n)$ ，必须谨慎使用，尽量避免。调用 pop 方法从头部弹出元素，性能也是很差，时间复杂度同样是$O(n)$。\n",
    "\n",
    "如果你需要频繁操作列表头部，可以考虑使用标准库里的 deque ，这是一种 双端队列 结构。 deque 头部和尾部操作性能都很优秀，时间复杂度都是$O(1)$。如果你需要一个 先进先出 ( FIFO )队列，可以这么写："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-28T08:00:16.055795Z",
     "start_time": "2021-03-28T08:00:16.033795Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from collections import deque\n",
    "\n",
    "q = deque()\n",
    "\n",
    "# enqueue\n",
    "q.append(1)\n",
    "\n",
    "# dequeue\n",
    "q.popleft()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 浅拷贝"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "调用 list 对象 copy 方法，可将列表拷贝一份，生成一个全新的列表："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-28T08:00:34.321638Z",
     "start_time": "2021-03-28T08:00:34.303638Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# 创建一个列表\n",
    ">>> l = [1, [2], 'three']\n",
    ">>> l\n",
    "[1, [2], 'three']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-28T08:00:38.079789Z",
     "start_time": "2021-03-28T08:00:38.071790Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# 拷贝形成一个新列表\n",
    ">>> l2 = l.copy()\n",
    ">>> l2\n",
    "[1, [2], 'three']\n",
    ">>> id(l), id(l2)\n",
    "(4417331976, 4420505736)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "print(id(l[0]),id(l2[0]))\n",
    "print(id(l[1][0]),id(l2[1][0]))\n",
    "print(id(l[2]),id(l2[2]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# 修改新列表不影响旧列表\n",
    ">>> l2[0] = 'one'\n",
    ">>> l2\n",
    "['one', [2], 'three']\n",
    ">>> l\n",
    "[1, [2], 'three']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "print(id(l[0]),id(l2[0]))\n",
    "print(id(l[1][0]),id(l2[1][0]))\n",
    "print(id(l[2]),id(l2[2]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# 但是修改新列表中的可变元素，旧列表也跟着变！\n",
    ">>> l2[1][0] = 'two'\n",
    ">>> l2\n",
    "['one', ['two'], 'three']\n",
    ">>> l\n",
    "[1, ['two'], 'three']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "print(id(l[0]),id(l2[0]))\n",
    "print(id(l[1][0]),id(l2[1][0]))\n",
    "print(id(l[2]),id(l2[2]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "由于 copy 方法只是对列表对象进行 浅拷贝 ，对新列表可变元素的修改对旧列表可见！\n",
    "\n",
    "如何理解浅拷贝呢？我们知道，列表对象内部数组保存元素对象的 指针；copy方法复制内部数组时，拷贝的也是元素对象的指针，而不是将元素对象拷贝一遍。因此，新列表对象与旧列表保存的都是同一组对象：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_31.jpg\" width=800 height=600>  \n",
    "\n",
    "deepcopy 函数将递归复制所有容器对象，确保新旧列表不会包含同一个容器对象。这样一来，代码第 13 行处的修改，便对原列表不可见了。深拷贝的行为跟浅拷贝恰好相反。\n",
    "\n",
    "浅拷贝 和 深拷贝 是 Python 面试中频繁考察的概念，必须完全掌握。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# 创建一个列表\n",
    "l = [1, [2], 'three']\n",
    "l\n",
    "# [1, [2], 'three']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# 对列表进行深拷贝\n",
    ">>> from copy import deepcopy\n",
    ">>> l2 = deepcopy(l)\n",
    ">>> l2\n",
    "[1, [2], 'three']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "print(id(l[0]),id(l2[0]))\n",
    "print(id(l[1][0]),id(l2[1][0]))\n",
    "print(id(l[2]),id(l2[2]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# 修改新列表不会影响旧列表\n",
    ">>> l2[1][0] = 'two'\n",
    ">>> l2\n",
    "[1, ['two'], 'three']\n",
    ">>> l\n",
    "[1, [2], 'three']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "print(id(l[0]),id(l2[0]))\n",
    "print(id(l[1][0]),id(l2[1][0]))\n",
    "print(id(l[2]),id(l2[2]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# 修改新列表不会影响旧列表\n",
    ">>> l2[0] = 'one'\n",
    ">>> l2\n",
    "[1, ['two'], 'three']\n",
    ">>> l\n",
    "[1, [2], 'three']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "print(id(l[0]),id(l2[0]))\n",
    "print(id(l[1][0]),id(l2[1][0]))\n",
    "print(id(l[2]),id(l2[2]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 总结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "本节我们一起回顾了 list 对象的典型用法并在此基础上研究其内部结构。 list 对象是一种 变长对象 ，内部结构除了变长对象 公共头部 外，维护着一个 动态数组 ，用于保存元素对象指针。其中，关键字段包括：\n",
    "\n",
    "ob_item ， 动态数组 指针，数组保存元素对象指针；\n",
    "allocated ，动态数组长度，即列表 容量 ；\n",
    "ob_size ，动态数组已保存元素个数，即列表 长度 。\n",
    "Python 内部负责管理 list 对象的容量，在必要时 自动扩缩容 ，极大降低开发人员的负担。\n",
    "\n",
    "列表头部操作与尾部操作的性能差距非常大，而collections.deque作为替代品可解决列表头部操作的性能问题。列表copy方法只实现了浅拷贝 ，想要深拷贝只能借助copy.deepcopy函数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## list 源码解析：动态数组精讲"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "list 对象是一种**容量自适应**的**线性容器**，底层由**动态数组**实现。动态数组结构决定了list对象具有优秀的尾部操作性能，但头部操作性能却很差劲。研发人员只有对底层数据结构有足够的认识，才能最大限度避免问题代码。\n",
    "\n",
    "现成的动态数组实现很多，除了我们正在研究的 list 对象，C++ 中的 vector 也是众所周知。虽然在实际项目中需要自行实现动态数组的场景已经很少很少了，但是源码还是有必要研究一番。源码研究不仅能加深对数据结构的理解，还能进一步提升编程水平，裨益颇多。\n",
    "\n",
    "本节，我们开始 list 对象源码，深入学习**动态数组**实现的艺术。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 容量调整"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "当我们调用 append 、pop 、insert 等方法时，列表长度随之发生变化。当列表长度超过底层数组容量时，便需要对底层数组进行**扩容**；当列表长度远低于底层数组容量时，便需要对底层数组进行**缩容**。\n",
    "\n",
    "Objects/listobject.c 源码表明，append 等方法依赖 list_resize 函数调整列表长度，扩容缩容的秘密就藏在这里！list_resize 函数在调整列表长度前，先检查底层数组容量，并在必要时重新分配底层数组。接下来，我们一起来解读 list_resize 函数，该函数同样位于源文件 Objects/listobject.c 中：\n",
    "\n",
    "    static int\n",
    "    list_resize(PyListObject *self, Py_ssize_t newsize)\n",
    "    {\n",
    "        PyObject **items;\n",
    "        size_t new_allocated, num_allocated_bytes;\n",
    "        Py_ssize_t allocated = self->allocated;\n",
    "\n",
    "        /* Bypass realloc() when a previous overallocation is large enough\n",
    "           to accommodate the newsize.  If the newsize falls lower than half\n",
    "           the allocated size, then proceed with the realloc() to shrink the list.\n",
    "        */\n",
    "        if (allocated >= newsize && newsize >= (allocated >> 1)) {\n",
    "            assert(self->ob_item != NULL || newsize == 0);\n",
    "            Py_SIZE(self) = newsize;\n",
    "            return 0;\n",
    "        }\n",
    "\n",
    "        /* This over-allocates proportional to the list size, making room\n",
    "         * for additional growth.  The over-allocation is mild, but is\n",
    "         * enough to give linear-time amortized behavior over a long\n",
    "         * sequence of appends() in the presence of a poorly-performing\n",
    "         * system realloc().\n",
    "         * The growth pattern is:  0, 4, 8, 16, 25, 35, 46, 58, 72, 88, ...\n",
    "         * Note: new_allocated won't overflow because the largest possible value\n",
    "         *       is PY_SSIZE_T_MAX * (9 / 8) + 6 which always fits in a size_t.\n",
    "         */\n",
    "        new_allocated = (size_t)newsize + (newsize >> 3) + (newsize < 9 ? 3 : 6);\n",
    "        if (new_allocated > (size_t)PY_SSIZE_T_MAX / sizeof(PyObject *)) {\n",
    "            PyErr_NoMemory();\n",
    "            return -1;\n",
    "        }\n",
    "\n",
    "        if (newsize == 0)\n",
    "            new_allocated = 0;\n",
    "        num_allocated_bytes = new_allocated * sizeof(PyObject *);\n",
    "        items = (PyObject **)PyMem_Realloc(self->ob_item, num_allocated_bytes);\n",
    "        if (items == NULL) {\n",
    "            PyErr_NoMemory();\n",
    "            return -1;\n",
    "        }\n",
    "        self->ob_item = items;\n",
    "        Py_SIZE(self) = newsize;\n",
    "        self->allocated = new_allocated;\n",
    "        return 0;\n",
    "    }\n",
    "    \n",
    "在函数开头，有几个局部变量定义，对理解函数逻辑非常关键：\n",
    "\n",
    "- items 指针，用于保存新数组；\n",
    "- new_allocated ，用于保存新数组容量；\n",
    "- num_allocated_bytes ，用于保存新数组内存大小，以字节为单位；\n",
    "- allocated ，用于保存旧数组容量。\n",
    "\n",
    "然后，代码第 12 行，检查新长度与底层数组容量的关系。如果新长度不超过数组容量，且大于等于数组容量的一半，则无需调整底层数组，直接更新 ob_size 字段。换句话讲， list 对象扩缩容的条件分别如下：\n",
    "\n",
    "- 扩容条件 ，新长度大于底层数组长度；\n",
    "- 缩容条件 ，新长度小于底层数组长度的一半；\n",
    "\n",
    "扩容或缩容条件触发时，list_resize 函数根据新长度计算数组容量并重新分配底层数组（第 27-44 行）：\n",
    "\n",
    "1. 第 27 行，新容量在长度加上 $\\frac{1}{8}$ 的裕量，再加上 3 或 6 的裕量；\n",
    "2. 第 28-31 行，如果新容量超过允许范围，返回错误；\n",
    "3. 第 33-34 行，如果新长度为 0 ，将新容量也设置为 0 ，因此空列表底层数组亦为空；\n",
    "4. 第 36-40 行，调用 PyMem_Realloc 函数重新分配底层数组；\n",
    "5. 第 41-44 行，更新 3 个关键字段，依次设置为 新底层数组 、 新长度 以及 新容量 。\n",
    "\n",
    "注意到代码第 27 行，新容量的计算公式有点令人费解。为什么还要加上 3 或者 6 的裕量呢？试想一下，如果新长度小于 8 ，那么$ \\frac{1}{8} $ 的裕量便是 0 ！这意味着，当 list 对象长度从 0 开始增长时，需要频繁扩容！\n",
    "\n",
    "为了解决这个问题，必须在 $\\frac{1}{8}$裕量的基础上额外加上一定的固定裕量。而 3 和 6 这两个特殊数值的选择，使得列表容量按照 0、4、8、 16、25、35、46、58、72、88…… 这样的序列进行扩张。这样一来，当 list 对象长度较小时，容量翻倍扩展，扩容频率得到有效限制。\n",
    "\n",
    "顺便提一下， PyMem_Realloc 函数是 Python 内部实现的内存管理函数之一，功能与 C 库函数 realloc 类似：\n",
    "\n",
    "    PyAPI_FUNC(void *) PyMem_Realloc(void *ptr, size_t new_size);\n",
    "\n",
    "PyMem_Realloc 函数用于对动态内存进行扩容或者缩容，关键步骤如下：\n",
    "\n",
    "1. 新申请一块尺寸为 new_size 的内存区域；\n",
    "2. 将数据从旧内存区域 ptr 拷贝到新内存区域；\n",
    "3. 释放旧内存区域 ptr ；\n",
    "4. 返回新内存区域。\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_32.png\" width=800 height=600>  \n",
    "\n",
    "内存管理 是最考验研发人员编程功底的领域之一，鼓励大家到 PyMem_Realloc 源码（./Objects/obmalloc.c）中进一步研究内存管理的技巧。学有余力的童鞋，可模仿着自己实现一个 realloc 函数，假以时日编程内功将突飞猛进！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 尾部追加"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "append 方法在 Python 内部由 C 函数 list_append 实现，而 list_append 进一步调用 app1 函数完成元素追加：\n",
    "\n",
    "    static int\n",
    "    app1(PyListObject *self, PyObject *v)\n",
    "    {\n",
    "        Py_ssize_t n = PyList_GET_SIZE(self);\n",
    "\n",
    "        assert (v != NULL);\n",
    "        if (n == PY_SSIZE_T_MAX) {\n",
    "            PyErr_SetString(PyExc_OverflowError,\n",
    "                \"cannot add more objects to list\");\n",
    "            return -1;\n",
    "        }\n",
    "\n",
    "        if (list_resize(self, n+1) < 0)\n",
    "            return -1;\n",
    "\n",
    "        Py_INCREF(v);\n",
    "        PyList_SET_ITEM(self, n, v);\n",
    "        return 0;\n",
    "    }\n",
    "    \n",
    "1. 第 4 行，调用 PyList_GET_SIZE 取出列表长度，即 ob_size 字段；\n",
    "2. 第 7-11 行，判断列表当前长度，如果已经达到最大限制，则报错；\n",
    "3. 第 13-15 行，调用 list_resize 更新列表长度，必要时 list_resize 对底层数组进行 扩容 ；\n",
    "4. 第 16 行，自增元素对象 引用计数 (元素对象新增一个来自列表对象的引用)；\n",
    "5. 第 17 行，将元素对象指针保存到列表最后一个位置，列表新长度为 n+1 ，最后一个位置下标为 n 。\n",
    "\n",
    "我们看到，有了 list_resize 这个辅助函数后， app1 函数的实现就非常直白了。接下来，我们将看到 insert、pop 等方法的实现中也用到这个函数，从中可体会到程序逻辑 划分 、 组合 的巧妙之处。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 头部插入"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "insert 方法在 Python 内部由 C 函数 list_insert_impl 实现，而 list_insert_impl 则调用 ins1 函数完成元素插入：\n",
    "\n",
    "    static int\n",
    "    ins1(PyListObject *self, Py_ssize_t where, PyObject *v)\n",
    "    {\n",
    "        Py_ssize_t i, n = Py_SIZE(self);\n",
    "        PyObject **items;\n",
    "        if (v == NULL) {\n",
    "            PyErr_BadInternalCall();\n",
    "            return -1;\n",
    "        }\n",
    "        if (n == PY_SSIZE_T_MAX) {\n",
    "            PyErr_SetString(PyExc_OverflowError,\n",
    "                \"cannot add more objects to list\");\n",
    "            return -1;\n",
    "        }\n",
    "\n",
    "        if (list_resize(self, n+1) < 0)\n",
    "            return -1;\n",
    "\n",
    "        if (where < 0) {\n",
    "            where += n;\n",
    "            if (where < 0)\n",
    "                where = 0;\n",
    "        }\n",
    "        if (where > n)\n",
    "            where = n;\n",
    "        items = self->ob_item;\n",
    "        for (i = n; --i >= where; )\n",
    "            items[i+1] = items[i];\n",
    "        Py_INCREF(v);\n",
    "        items[where] = v;\n",
    "        return 0;\n",
    "    }\n",
    "\n",
    "1. 第 4 行，调用 PyList_GET_SIZE 取出列表长度，即 ob_size 字段；\n",
    "2. 第 10-14 行，判断列表当前长度，如果已经达到最大限制，则报错；\n",
    "3. 第 16-17 行，调用 list_resize 更新列表长度，必要时 list_resize 对底层数组进行 扩容 ；\n",
    "4. 第 19-23 行，检查插入位置下标，如果下标为负数，加上 n 将其转换为非负数；\n",
    "5. 第 21-22、24-25 行，检查插入位置下标是否越界，如果越界则设为开头或结尾；\n",
    "6. 第 26-28 行，将插入位置以后的所有元素逐一往后移一个位置，特别注意 for 循环必须 从后往前 迭代；\n",
    "7. 第 29 行，自增元素对象 引用计数 (元素对象新增一个来自列表对象的引用)；\n",
    "8. 第 30 行，将元素对象指针保存到列表指定位置。\n",
    "\n",
    "Python 序列 下标很有特色 ，除了支持 0~n-1 这样的惯例外，还支持 倒数下标 。倒数下标为负数，从后往前数：最后一个元素为 -1 ，倒数第二个为 -2 ；以此类推，第一个元素下标为： -n 。\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_33.png\" width=800 height=600>  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "insert 方法在 Python 内部由 C 函数 list_insert_impl 实现，而 list_insert_impl 则调用 ins1 函数完成元素插入：\n",
    "\n",
    "    static int\n",
    "    ins1(PyListObject *self, Py_ssize_t where, PyObject *v)\n",
    "    {\n",
    "        Py_ssize_t i, n = Py_SIZE(self);\n",
    "        PyObject **items;\n",
    "        if (v == NULL) {\n",
    "            PyErr_BadInternalCall();\n",
    "            return -1;\n",
    "        }\n",
    "        if (n == PY_SSIZE_T_MAX) {\n",
    "            PyErr_SetString(PyExc_OverflowError,\n",
    "                \"cannot add more objects to list\");\n",
    "            return -1;\n",
    "        }\n",
    "\n",
    "        if (list_resize(self, n+1) < 0)\n",
    "            return -1;\n",
    "\n",
    "        if (where < 0) {\n",
    "            where += n;\n",
    "            if (where < 0)\n",
    "                where = 0;\n",
    "        }\n",
    "        if (where > n)\n",
    "            where = n;\n",
    "        items = self->ob_item;\n",
    "        for (i = n; --i >= where; )\n",
    "            items[i+1] = items[i];\n",
    "        Py_INCREF(v);\n",
    "        items[where] = v;\n",
    "        return 0;\n",
    "    }\n",
    "\n",
    "1. 第 4 行，调用 PyList_GET_SIZE 取出列表长度，即 ob_size 字段；\n",
    "2. 第 10-14 行，判断列表当前长度，如果已经达到最大限制，则报错；\n",
    "3. 第 16-17 行，调用 list_resize 更新列表长度，必要时 list_resize 对底层数组进行 扩容 ；\n",
    "4. 第 19-23 行，检查插入位置下标，如果下标为负数，加上 n 将其转换为非负数；\n",
    "5. 第 21-22、24-25 行，检查插入位置下标是否越界，如果越界则设为开头或结尾；\n",
    "6. 第 26-28 行，将插入位置以后的所有元素逐一往后移一个位置，特别注意 for 循环必须 从后往前 迭代；\n",
    "7. 第 29 行，自增元素对象 引用计数 (元素对象新增一个来自列表对象的引用)；\n",
    "8. 第 30 行，将元素对象指针保存到列表指定位置。\n",
    "\n",
    "Python 序列 下标很有特色 ，除了支持 0~n-1 这样的惯例外，还支持 倒数下标 。倒数下标为负数，从后往前数：最后一个元素为 -1 ，倒数第二个为 -2 ；以此类推，第一个元素下标为： -n 。\n",
    "\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_31.png\" width=800 height=600>  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 尾部追加"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "append 方法在 Python 内部由 C 函数 list_append 实现，而 list_append 进一步调用 app1 函数完成元素追加：\n",
    "\n",
    "    static int\n",
    "    app1(PyListObject *self, PyObject *v)\n",
    "    {\n",
    "        Py_ssize_t n = PyList_GET_SIZE(self);\n",
    "\n",
    "        assert (v != NULL);\n",
    "        if (n == PY_SSIZE_T_MAX) {\n",
    "            PyErr_SetString(PyExc_OverflowError,\n",
    "                \"cannot add more objects to list\");\n",
    "            return -1;\n",
    "        }\n",
    "\n",
    "        if (list_resize(self, n+1) < 0)\n",
    "            return -1;\n",
    "\n",
    "        Py_INCREF(v);\n",
    "        PyList_SET_ITEM(self, n, v);\n",
    "        return 0;\n",
    "    }\n",
    "    \n",
    "1. 第 4 行，调用 PyList_GET_SIZE 取出列表长度，即 ob_size 字段；\n",
    "2. 第 7-11 行，判断列表当前长度，如果已经达到最大限制，则报错；\n",
    "3. 第 13-15 行，调用 list_resize 更新列表长度，必要时 list_resize 对底层数组进行 扩容 ；\n",
    "4. 第 16 行，自增元素对象 引用计数 (元素对象新增一个来自列表对象的引用)；\n",
    "5. 第 17 行，将元素对象指针保存到列表最后一个位置，列表新长度为 n+1 ，最后一个位置下标为 n 。\n",
    "\n",
    "我们看到，有了 list_resize 这个辅助函数后， app1 函数的实现就非常直白了。接下来，我们将看到 insert、pop 等方法的实现中也用到这个函数，从中可体会到程序逻辑 划分 、 组合 的巧妙之处。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 弹出元素"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "pop 方法将指定下标的元素从列表中弹出，下标默认为 -1 。换句话讲，如果未指定下标，pop 弹出最后一个元素，pop 方法在 Python 内部由 C 函数 list_pop_impl 实现：\n",
    "\n",
    "    static PyObject *\n",
    "    list_pop_impl(PyListObject *self, Py_ssize_t index)\n",
    "    {\n",
    "        PyObject *v;\n",
    "        int status;\n",
    "\n",
    "        if (Py_SIZE(self) == 0) {\n",
    "            /* Special-case most common failure cause */\n",
    "            PyErr_SetString(PyExc_IndexError, \"pop from empty list\");\n",
    "            return NULL;\n",
    "        }\n",
    "        if (index < 0)\n",
    "            index += Py_SIZE(self);\n",
    "        if (index < 0 || index >= Py_SIZE(self)) {\n",
    "            PyErr_SetString(PyExc_IndexError, \"pop index out of range\");\n",
    "            return NULL;\n",
    "        }\n",
    "        v = self->ob_item[index];\n",
    "        if (index == Py_SIZE(self) - 1) {\n",
    "            status = list_resize(self, Py_SIZE(self) - 1);\n",
    "            if (status >= 0)\n",
    "                return v; /* and v now owns the reference the list had */\n",
    "            else\n",
    "                return NULL;\n",
    "        }\n",
    "        Py_INCREF(v);\n",
    "        status = list_ass_slice(self, index, index+1, (PyObject *)NULL);\n",
    "        if (status < 0) {\n",
    "            Py_DECREF(v);\n",
    "            return NULL;\n",
    "        }\n",
    "        return v;\n",
    "    }\n",
    "\n",
    "\n",
    "1. 第 7-11 行，如果列表为空，没有任何元素可弹出，抛出 IndexError 异常；\n",
    "2. 第 12-13 行，如果给定下标为**倒数下标**，先加上列表长度，将其转换成普通下标；\n",
    "3. 第 14-16 行，检查给定下标是否在合法范围内，超出合法范围同样抛出 IndexError 异常；\n",
    "4. 第 18 行，从底层数组中取出待弹出元素；\n",
    "5. 第 19-25 行，如果待弹出元素为列表最后一个，调用 list_resize 快速调整列表长度即可，无需移动其他元素；\n",
    "6. 第 26-31 行，其他情况下调用 list_ass_slice 函数删除元素，调用前需要通过 Py_INCREF 增加元素引用计数，因为 list_ass_slice 函数内部将释放被删除元素；\n",
    "7. 第 32 行，将待弹出元素返回。\n",
    "\n",
    "list_ass_slice 函数其实有两种不同的语义，具体执行哪种语义由函数参数决定，函数接口如下：\n",
    "\n",
    "    /* a[ilow:ihigh] = v if v != NULL.\n",
    "     * del a[ilow:ihigh] if v == NULL.\n",
    "     *\n",
    "     * Special speed gimmick:  when v is NULL and ihigh - ilow <= 8, it's\n",
    "     * guaranteed the call cannot fail.\n",
    "     */\n",
    "    static int\n",
    "    list_ass_slice(PyListObject *a, Py_ssize_t ilow, Py_ssize_t ihigh, PyObject *v);\n",
    "    \n",
    "- 替换语义 ，如果最后一个参数 v 值不为 NULL ，执行替换语义，即 a[ilow:ihigh] = v 。\n",
    "- 删除语义 ，如果最后一个参数 v 值为 NULL ，执行删除语义，即：del a[ilow:ihigh] ；\n",
    "\n",
    "因此，代码第 27 行中， list_ass_slice 函数执行删除语义，将 $[index, index+1)$ 范围内的元素删除。由于半开半闭区间 $[index, index+1)$ 中只包含 index 一个元素，效果等同于将下标为 index 的元素删除。\n",
    "\n",
    "执行删除语义时， list_ass_slice 函数将被删元素后面的元素逐一往前移动，以便重新覆盖删除操作所造成的空隙。由此可见，pop 方法弹出元素，时间复杂度跟弹出位置有关：\n",
    "\n",
    "- 最好时间复杂度 ( 尾部弹出 )，$O(1)$ ；\n",
    "- 最坏时间复杂度 ( 头部弹出 )，$O(n)$；\n",
    "- 平均时间复杂度， $O(\\frac{n}{2})$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-27T02:37:38.688845Z",
     "start_time": "2021-03-27T02:37:38.675843Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "help(list.pop)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 删除元素"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "remove 方法将给定元素从列表中删除。与 pop 略微不同，remove 方法直接给定待删除元素，而不是元素下标。remove 方法在 Python 内部由 C 函数 list_remove 实现：\n",
    "\n",
    "    static PyObject *\n",
    "    list_remove(PyListObject *self, PyObject *value)\n",
    "    /*[clinic end generated code: output=f087e1951a5e30d1 input=2dc2ba5bb2fb1f82]*/\n",
    "    {\n",
    "        Py_ssize_t i;\n",
    "\n",
    "        for (i = 0; i < Py_SIZE(self); i++) {\n",
    "            int cmp = PyObject_RichCompareBool(self->ob_item[i], value, Py_EQ);\n",
    "            if (cmp > 0) {\n",
    "                if (list_ass_slice(self, i, i+1,\n",
    "                                   (PyObject *)NULL) == 0)\n",
    "                    Py_RETURN_NONE;\n",
    "                return NULL;\n",
    "            }\n",
    "            else if (cmp < 0)\n",
    "                return NULL;\n",
    "        }\n",
    "        PyErr_SetString(PyExc_ValueError, \"list.remove(x): x not in list\");\n",
    "        return NULL;\n",
    "    }\n",
    "    \n",
    "list_remove 函数先遍历列表中每个元素（第 7 行），检查元素是否为待删除元素 value （第 8 行），以此确定下标。然后， list_remove 函数调用 list_ass_slice 函数进行删除。注意到，如果给定元素不存在， list_remove 将抛出 ValueError 异常。\n",
    "\n",
    "由此可见，remove 方法在删除前有一个时间复杂度为 O(n)O(n) 的查找过程，性能不甚理想，须谨慎使用。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 总结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "list 对象是一种**容量自适应**的**线性容器**，底层由**动态数组**实现。Python 内部由函数 list_resize 调整列表长度， list_resize 自动为列表进行**扩容**或者**缩容**：\n",
    "\n",
    "- 底层数组容量不够时，需要进行**扩容**；\n",
    "- 扩容时， Python 额外分配大约 1/8的容量裕量，以控制扩容频率；\n",
    "- 底层数组空闲位置超过一半时，需要进行**缩容**。\n",
    "\n",
    "动态数组的特性决定了 list 对象相关操作性能有好有坏，使用时须特别留意：\n",
    "\n",
    "- append 向尾部追加元素，时间复杂度为$O(1)$ ，放心使用；\n",
    "- insert 往列表插入元素，最坏时间复杂度是$O(n)$，平均时间复杂度也是$O(n)$，须谨慎使用；\n",
    "- pop 从列表中弹出元素，最好时间复杂度为$O(1)$，平均时间复杂度为$O(n)$ ，弹出非尾部元素时需谨慎；\n",
    "remove 从列表中删除元素，时间复杂度为 O(n)O(n)，同样须谨慎使用。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## dict 对象，高效的关联式容器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Python 中的 dict 对象是一种**关联式容器**对象，用于保存由**键**( key )到**值**( value )的映射关系。借助关联式容器，程序可快速定位到与指定**键**相关联的**值**。dict 对象在 Python 程序中使用频率非常高，如果应用不当将严重影响程序的执行效率。\n",
    "\n",
    "本节，我们先从 dict 对象常用操作入手，回顾它的**基本用法**；接着结合其他内建容器对象，研究 dict 关键操作的**执行效率**；最后以 dict 对象**内部结构**收尾，详细讲解其内部**哈希表**的实现要点，以及其中几个关键**性能考量**。相信通过本节学习，读者将对 dict 实现原理了如指掌，这对用好 dict 对象至关重要。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 基本用法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-27T03:05:28.895813Z",
     "start_time": "2021-03-27T03:05:28.884805Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# 我们用一个 dict 对象来保存培训班学员的成绩，先创建一个空对象：\n",
    "scores = {}\n",
    "scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "那么，一个什么都不放的 dict 对象需要占用多少内存呢？根据前面章节，我们知道对象头部字段是必不可少的。可我们很惊讶地发现，一个空的 dict 对象居然要占用 240 字节的内存！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-27T03:05:54.717402Z",
     "start_time": "2021-03-27T03:05:54.711419Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.getsizeof(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "现在将 jim 的成绩保存保存到 dict 对象中："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-27T03:06:56.120033Z",
     "start_time": "2021-03-27T03:06:56.113038Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "scores['jim'] = 70\n",
    "sys.getsizeof(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "数据插入后，我们发现 dict 对象内存使用量保存不变。看来， dict 对象也有一种类似 list 对象的 预分配机制 。\n",
    "\n",
    "现在，接着存入 lily 、lucy 以及 tom 的成绩。我们发现， dict 还没达到扩容条件，内存大小保存不变："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-27T03:07:48.605178Z",
     "start_time": "2021-03-27T03:07:48.598181Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "scores['lily'] = 75\n",
    "scores['lucy'] = 80\n",
    "scores['tom'] = 90\n",
    "scores['alice'] = 95\n",
    "sys.getsizeof(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "”快速“不是一个精确的形容词，到底多快呢？这里先给出答案，由于 dict 对象底层由哈希表实现， 查找操作平均时间复杂度是$O(1)$。当然了，在哈希不均匀的情况下，最坏时间复杂度是$O(n)$，但一般情况下很难发生。\n",
    "\n",
    "当然了，如果有某位学员(例如 lily)转学了，可通过 pop 方法将其剔除："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-27T03:08:54.860949Z",
     "start_time": "2021-03-27T03:08:54.848955Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "scores.pop('lily')\n",
    "scores, sys.getsizeof(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "哈希表结构决定了 dict 的删除操作也很快，平均时间复杂度也是$O(1)$。实际上， dict 插入、删除、查找的平均时间复杂度都是 ，$O(1)$最坏时间复杂度是$O(n)$。因此，哈希函数的选择就至关重要，一个好的哈希函数应该将键尽可能**均匀**地映射到哈希空间中，最大限度地避免**哈希冲突**。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 执行效率"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "我们知道 dict 对象搜索操作时间复杂度为$O(1)$，远远好于 list 对象的$O(n)$。这意味着什么了？为得到一个更准确、直观的感受，我们编写一个测试程序，分别测试不同规模 dict 、 list 对象完成 1000 次搜索所需的时间："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-27T03:11:21.927548Z",
     "start_time": "2021-03-27T03:10:59.598266Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "import random\n",
    "import time\n",
    "\n",
    "# 随机数生成器\n",
    "randint = lambda: random.randint(-2**30+1, 2**30-1)\n",
    "\n",
    "def count_targets(items, targets):\n",
    "    '''\n",
    "    计算目标对象出现个数\n",
    "        items: 待搜索容器\n",
    "        targets: 待搜索目标元素列表\n",
    "    '''\n",
    "\n",
    "    found = 0\n",
    "\n",
    "    for target in targets:\n",
    "        if target in items:\n",
    "            found += 1\n",
    "\n",
    "    return found\n",
    "\n",
    "def generate_random_dict(n):\n",
    "    '''\n",
    "    生成随机数字典\n",
    "    '''\n",
    "\n",
    "    dict_items = {}\n",
    "    while len(dict_items) < n:\n",
    "        dict_items[randint()] = True\n",
    "\n",
    "    return dict_items\n",
    "\n",
    "def generate_random_list(n):\n",
    "    '''\n",
    "    生成随机数列表\n",
    "    '''\n",
    "\n",
    "    return [\n",
    "        randint()\n",
    "        for _ in range(0, n)\n",
    "    ]\n",
    "\n",
    "def test_for_scale(scale, targets):\n",
    "    '''\n",
    "    执行一个样例\n",
    "        scale: 测试容器规模\n",
    "        targets: 待搜索元素列表\n",
    "    '''\n",
    "\n",
    "    # 生成指定规模的随机数容器\n",
    "    dict_items = generate_random_dict(scale)\n",
    "    list_items = generate_random_list(scale)\n",
    "\n",
    "    # 测试dict搜索所需时间\n",
    "    start_ts = time.time()\n",
    "    count_targets(dict_items, targets)\n",
    "    dict_time = time.time() - start_ts\n",
    "\n",
    "    # 测试list搜索所需时间\n",
    "    start_ts = time.time()\n",
    "    count_targets(list_items, targets)\n",
    "    list_time = time.time() - start_ts\n",
    "\n",
    "    # 打印结果\n",
    "    print('Scale:', scale)\n",
    "    print('Dict:', dict_time)\n",
    "    print('List:', list_time)\n",
    "    print()\n",
    "\n",
    "def main():\n",
    "    # 每次测试搜索1000次\n",
    "    # 生成1000个随机数作为搜索目标\n",
    "    targets = generate_random_list(1000)\n",
    "\n",
    "    # 以不同规模运行测试函数\n",
    "    for scale in [1000, 10000, 100000, 1000000]:\n",
    "        test_for_scale(scale, targets)\n",
    "        \n",
    "main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "我们将测试结果制作表格， dict 和 list 的表现一目了然：\n",
    "\n",
    "|容器规模|增长系数|dict消耗时间|dict增长系数|list消耗时间|list增长系数|\n",
    "|:-:|:-:|:-:|:-:|:-:|:-:|\n",
    "|1000|\t1|\t0.000129s|\t1\t|0.036s|\t1|\n",
    "|10000|\t10|\t0.000172s|\t1.33|\t0.348s|\t9.67|\n",
    "|100000|\t100|\t0.000216s|\t1.67|\t3.679s|\t102.19|\n",
    "|1000000|\t1000|\t0.000382s|\t2.96|\t48.044s|\t1335.56|\n",
    "\n",
    "从表格中，我们看到：当容器规模增长 1000 倍， dict 搜索时间几乎保持不变，但 list 搜索时间增长了差不多 1000 倍。当规模达到 10 万时，1000 次 list 搜索花了接近一分钟时间，而 dict 只需 382 微秒！dict 对象完成一次搜索只需 0.382 微秒，也就是说一秒钟可以完成 200 多万次搜索！\n",
    "\n",
    "dict 对象到底用了什么黑科技呢？接下来，我们一起从它的内部结构中寻找答案。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 内部结构"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "由于关联式容器使用场景非常广泛，几乎所有现代编程语言都提供某种关联式容器，而且特别关注键的$搜索效率$。例如， C++ 标准模板库中的 map 就是一种关联式容器，内部基于$红黑树$实现。红黑树是一种$平衡$二叉树，能够提供良好的操作效率，插入、删除、搜索等关键操作的时间复杂度均为$O(log_2 n)$。\n",
    "\n",
    "Python 虚拟机的运行重度依赖 dict 对象，包括$名字空间$以及$对象属性空间$等概念底层都是由 dict 对象实现的。因此， Python 对 dict 对象的效率要求更为苛刻。那么，操作效率优于$O(log_2n)$ 的数据结构有哪些呢？好吧，你可能已经猜到了， Python 中的 dict 对象就是基于$散列表$实现的。\n",
    "\n",
    "现在，是时候揭开 dict 对象神秘的面纱了。 dict 对象在 Python 内部由结构体 PyDictObject 表示， PyDictObject 在头文件 Include/dictobject.h 中定义：\n",
    "\n",
    "    typedef struct {\n",
    "        PyObject_HEAD\n",
    "\n",
    "        /* Number of items in the dictionary */\n",
    "        Py_ssize_t ma_used;\n",
    "\n",
    "        /* Dictionary version: globally unique, value change each time\n",
    "           the dictionary is modified */\n",
    "        uint64_t ma_version_tag;\n",
    "\n",
    "        PyDictKeysObject *ma_keys;\n",
    "\n",
    "        /* If ma_values is NULL, the table is \"combined\": keys and values\n",
    "           are stored in ma_keys.\n",
    "\n",
    "           If ma_values is not NULL, the table is splitted:\n",
    "           keys are stored in ma_keys and values are stored in ma_values */\n",
    "        PyObject **ma_values;\n",
    "    } PyDictObject;\n",
    "    \n",
    "dict 对象理论上应该是一种变长对象，但 PyObject_HEAD 头部告诉我们， Python 其实把它作为普通对象实现。除了对象公共头部外， PyDictObject 还包括以下几个字段：\n",
    "\n",
    "- ma_used ，对象当前所保存的 键值对个数 ；\n",
    "- ma_version_tag ，对象当前 版本号 ，每次修改时更新；\n",
    "- ma_keys ，指向按键对象映射的 哈希表 结构；\n",
    "- ma_values ， 分离模式下指向由所有 值对象 组成的数组。\n",
    "\n",
    "到目前为止，我们还没找到哈希表的具体结构，但是已经发现了一些蛛丝马迹—— PyDictKeysObject 。现在我们趁热打铁，扒开 PyDictKeysObject 的源码看一看，Objects/dict-common.h 头文件中 _dictkeysobject 是也：\n",
    "\n",
    "    struct _dictkeysobject {\n",
    "        Py_ssize_t dk_refcnt;\n",
    "\n",
    "        /* Size of the hash table (dk_indices). It must be a power of 2. */\n",
    "        Py_ssize_t dk_size;\n",
    "\n",
    "        /* Function to lookup in the hash table (dk_indices):\n",
    "\n",
    "           - lookdict(): general-purpose, and may return DKIX_ERROR if (and\n",
    "             only if) a comparison raises an exception.\n",
    "\n",
    "           - lookdict_unicode(): specialized to Unicode string keys, comparison of\n",
    "             which can never raise an exception; that function can never return\n",
    "             DKIX_ERROR.\n",
    "\n",
    "           - lookdict_unicode_nodummy(): similar to lookdict_unicode() but further\n",
    "             specialized for Unicode string keys that cannot be the <dummy> value.\n",
    "\n",
    "           - lookdict_split(): Version of lookdict() for split tables. */\n",
    "        dict_lookup_func dk_lookup;\n",
    "\n",
    "        /* Number of usable entries in dk_entries. */\n",
    "        Py_ssize_t dk_usable;\n",
    "\n",
    "        /* Number of used entries in dk_entries. */\n",
    "        Py_ssize_t dk_nentries;\n",
    "\n",
    "        /* Actual hash table of dk_size entries. It holds indices in dk_entries,\n",
    "           or DKIX_EMPTY(-1) or DKIX_DUMMY(-2).\n",
    "\n",
    "           Indices must be: 0 <= indice < USABLE_FRACTION(dk_size).\n",
    "\n",
    "           The size in bytes of an indice depends on dk_size:\n",
    "\n",
    "           - 1 byte if dk_size <= 0xff (char*)\n",
    "           - 2 bytes if dk_size <= 0xffff (int16_t*)\n",
    "           - 4 bytes if dk_size <= 0xffffffff (int32_t*)\n",
    "           - 8 bytes otherwise (int64_t*)\n",
    "\n",
    "           Dynamically sized, SIZEOF_VOID_P is minimum. */\n",
    "        char dk_indices[];  /* char is required to avoid strict aliasing. */\n",
    "\n",
    "        /* \"PyDictKeyEntry dk_entries[dk_usable];\" array follows:\n",
    "           see the DK_ENTRIES() macro */\n",
    "    };\n",
    "    \n",
    "_dictkeysobject 结构体包含 dict 对象哈希表实现的所有秘密，结合注释可以解读其中的关键字段：\n",
    "\n",
    "- dk_refcnt ，引用计数，跟 映射视图 的实现有关，有点类似对象引用计数；\n",
    "- dk_size ，哈希表大小，必须是$2^n$，这样可将模运算优化成 按位与 运算；\n",
    "- dk_lookup ，**哈希查找函数**指针，可根据 dict 当前状态选用最优函数版本；\n",
    "- dk_usable ，键值对数组**可用个数**；\n",
    "- dk_nentries ，键值对数组**已用个数**；\n",
    "- dk_indices ，哈希表**起始地址**，哈希表后紧接着**键值对数组**dk_entries 。\n",
    "\n",
    "键值对结构体 PyDictKeyEntry 就非常直白了，除了保存键对象和值对象的指针外，缓存着键对象的哈希值：\n",
    "\n",
    "    typedef struct {\n",
    "        /* Cached hash code of me_key. */\n",
    "        Py_hash_t me_hash;\n",
    "        PyObject *me_key;\n",
    "        PyObject *me_value; /* This field is only meaningful for combined tables */\n",
    "    } PyDictKeyEntry;\n",
    "    \n",
    "- me_hash ，键对象的 哈希值 ，避免重复调用 __hash__ 计算哈希值；\n",
    "- me_key ，键对象指针；\n",
    "- me_value ，值对象指针。\n",
    "\n",
    "至此， dict 对象内部的哈希表结构已经非常清晰了：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_34.png\" width=800 height=600>  \n",
    "\n",
    "dict 对象真正的实现藏身于 PyDictKeysObject 中，它内部包含两个关键数组，一个是**键值对数组**dk_entries ，另一个是**哈希索引数组**dk_indices 。dict 所维护的键值对，按照先来后到的顺序保存于键值对数组中；而哈希索引数组对应槽位则保存着键值对在数组中的位置。\n",
    "\n",
    "如上图，当我们往空 dict 对象插入新键值对 (‘jim’, 70) 时， Python 执行以下步骤：\n",
    "\n",
    "1. 将键值对保存于 dk_entries 数组末尾，即下标为 0 的位置；\n",
    "2. 计算键对象 ‘jim’ 的哈希值并取右 3 位，得到该键在哈希索引数组中的下标 5 ；\n",
    "3. 将键值对在数组中的下标 0 ，保存于哈希索引数组中编号为 5 的槽位中。\n",
    "\n",
    "这样一来，查找操作便可快速进行，分为以下几个步骤：\n",
    "\n",
    "1. 计算键对象 ‘jim’ 的哈希值并取右 3 位，得到该键在哈希索引数组中的下标 5 ；\n",
    "2. 找到哈希索引数组下标为 5 的槽位，取出其中保存的下标 0 ；\n",
    "3. 找到键值对数组第 0 个位置，并取出 值对象 ；\n",
    "\n",
    "由于**哈希值计算**以及**数组定位**均可在常数时间内完成，以上操作均可在常数时间内完成，也就是**O(1)**。为简化讨论，Python 应对 哈希冲突 的策略我们先按下不表，留在下节 源码精讲 中详细展开。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 容量策略"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "根据行为我们断定， dict 对象也有一种类似 list 对象的 预分配机制 。那么， dict 对象容量管理策略是怎样的呢？\n",
    "\n",
    "由 Objects/dictobject.c 源文件中的 PyDict_MINSIZE 宏定义，我们知道 dict 内部哈希表最小长度为 8 ：\n",
    "\n",
    "    #define PyDict_MINSIZE 8\n",
    "\n",
    "哈希表越密集，哈希冲突则越频繁，性能也就越差。因此，哈希表必须是一种**稀疏**的表结构，越稀疏则性能越好。由于 内存开销 的制约，哈希表不可能无限度稀疏，需要在时间和空间上进行权衡。实践经验表明，一个$\\frac{1}{2}$到 $\\frac{2}{3}$ 满的哈希表，性能较为理想——以相对合理的 内存 换取相对高效的 执行性能 。\n",
    "\n",
    "为保证哈希表的稀疏程度，进而控制哈希冲突频率， Python 通过 USABLE_FRACTION 宏将哈希表内元素控制在 $\\frac{2}{3}$ \n",
    "以内。USABLE_FRACTION 宏根据哈希表规模 n ，计算哈希表可存储元素个数，也就是**键值对数组**的长度。以长度为 8 的哈希表为例，最多可以保持 5 个键值对，超出则需要扩容。USABLE_FRACTION 是一个非常重要的宏定义，位于源文件 Objects/dictobject.c 中：\n",
    "\n",
    "    #define USABLE_FRACTION(n) (((n) << 1)/3)\n",
    "\n",
    "哈希表规模一定是$2^n$,也就是说 Python 采用 翻倍扩容 策略。例如，长度为 8 的哈希表扩容后，长度变为16。\n",
    "\n",
    "最后，我们来考察一个空的 dict 对象所占用的内存空间。Python 为空 dict 对象分配了一个长度为 8 的哈希表，因而也要占用相当多的内存，主要有以下几个部分组成：\n",
    "\n",
    "- 可收集对象链表节点，共 24 字节，在此不再展开， 垃圾回收机制 讲解中有详细介绍；\n",
    "- PyDictObject 结构体，6 个字段，共 48 字节；\n",
    "- PyDictKeysObject 结构体，除两个数组外有 5 个字段，共 40 字节；\n",
    "- 哈希索引数组，长度为 8 ，每个槽位 1 字节，共 8 字节；\n",
    "- 键值对数组，长度为 5 ，每个 PyDictKeyEntry 结构体 24 字节，共 120 字节。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-27T04:25:23.737746Z",
     "start_time": "2021-03-27T04:25:23.731750Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    " sys.getsizeof({})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 内存优化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "在 Python 早期，哈希表并没有分成两个数组实现，而是由一个键值对数组实现，这个数组也承担哈希索引的角色：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_35.png\" width=800 height=600>  \n",
    "\n",
    "如上图，由一个键值对数组充当哈希表，哈希值直接在数组中定位到键值对，无须分成两个步骤，似乎更好。那么， Python 为啥多此一举将哈希表实现分为两个数组来实现呢？答案是出于**内存**考量。\n",
    "\n",
    "由于哈希表必须保持 稀疏 ，最多只有 $\\frac{2}{3}$ 满，这意味着要浪费至少$\\frac{1}{3}$的内存空间。更雪上加霜的是，一个键值对条目 PyDictKeyEntry 大小达 24 字节，试想一个规模为 65536 的哈希表，将浪费高达 512KB 的内存空间：\n",
    "\n",
    "为了尽量节省内存， Python 将键值对数组压缩到原来的$\\frac{2}{3}$，只负责存储，索引由另一个数组负责。由于索引数组只需要保存**键值对数组**的下标，而整数占用的内存空间只是若干字节，因此可以节约大量内存。\n",
    "\n",
    "**索引数组**可根据哈希表规模，选择尽量小的整数类型。对于规模不超过 256 的哈希表，选择 8 位整数即可；对于长度不超过 65536 的哈希表，16 位整数足矣；其他以此类推。\n",
    "\n",
    "以长度为 8 的哈希表为例，键值对数组只需分配 5 个单元，索引数组每个单元只需 1 字节即可：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_36.png\" width=800 height=600>  \n",
    "\n",
    "这样一来，以区区 8 字节的代价，挽回了 72 字节的开销，还是节约了 64 字节！虽然 64 字节不足挂齿，但对于规模较大的哈希表，节约下来的内存还是相当可观的：\n",
    "\n",
    "|哈希表规模|\t条目表规模\t|旧方案\t|新方案\t|节约内存|\n",
    "|:-:|:-:|:-:|:-:|:-:|\n",
    "|8|\t8 * 2 / 3 = 5|\t24 * 8 = 192|\t1 * 8 + 24 * 5 = 128|\t64|\n",
    "|256|\t256 * 2 / 3 = 170|\t24 * 256 = 6144|\t1 * 256 + 24 * 170 = 4336|\t1808|\n",
    "|65536|\t65536 * 2 / 3 = 43690|\t24 * 65536 = 1572864|\t2 * 65536 + 24 * 43690 = 1179632|393232|\n",
    "\n",
    "由此可见，Python 作者们为节约内存可谓是殚精竭虑，以后在面试中又有一个不错的谈资！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 总结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "本节，我们考察了 dict 对象的搜索效率，并深入源码研究其内部**哈希表**实现，收获颇多：\n",
    "\n",
    "1. dict 是一种高效的关联式容器，每秒完成高达 200 多万次搜索操作；\n",
    "2. dict 内部由哈希表实现，哈希表的 稀疏 特性意味着昂贵的内存开销；\n",
    "3. 为优化内存使用， Python 将 dict 哈希表分为 哈希索引 和 键值对 两个数组来实现；\n",
    "4. 哈希表在 $\\frac{1}{2}$  到 $\\frac{2}{3}$  满时，性能较为理想，较好地平衡了 内存开销 与 搜索效率 ；"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## dict 哈希表高级知识精讲"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "上一小节，我们通过源码学习，研究了 dict 对象的内部结构，并找到隐藏其中的秘密——**哈希表**。**关联式容器**一般由**平衡搜索树**或**哈希表**来实现，dict 选用**哈希表**，主要考虑**搜索效率**。但哈希表**稀疏**的特性，意味着巨大的内存开销。为优化内存使用，Python 别出心裁地将哈希表分成两部分来实现：**哈希索引**以及**键值对存储**数组。\n",
    "\n",
    "尽管如此，由于篇幅关系，很多细节我们还没来得及讨论。本节，我们再接再厉，继续研究**哈希函数**、**哈希冲突**、**哈希攻击**以及**删除操作**等高级知识点，彻底掌握哈希表设计精髓。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 哈希值"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Python 内置函数 hash 返回对象**哈希值**,**哈希表**依赖**哈希值**索引元素：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_36.png\" width=800 height=600>  \n",
    "\n",
    "根据哈希表性质， 键对象 必须满足以下两个条件，否则哈希表便不能正常工作：\n",
    "\n",
    "- 哈希值在对象整个生命周期内不能改变；\n",
    "- 可比较，且比较相等的对象哈希值必须相同；\n",
    "\n",
    "满足这两个条件的对象便是 **可哈希( hashable)** 对象，只有可哈希对象才可作为哈希表的键。因此，诸如 dict 、set等底层由哈希表实现的容器对象，其键对象必须是可哈希对象。\n",
    "\n",
    "Python 内建对象中的**不可变对象** ( immutable )都是可哈希对象；而诸如 list 、dict 等 可变对象 则不是："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-27T04:41:29.660366Z",
     "start_time": "2021-03-27T04:41:29.519433Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "hash([])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "不可哈希对象不能作为 dict 对象的键，显然 list 、 dict 等均不是合法的键对象：\n",
    "\n",
    "    >>> {\n",
    "    ...     []: 'list is not hashable'\n",
    "    ... }\n",
    "    Traceback (most recent call last):\n",
    "      File \"<stdin>\", line 2, in <module>\n",
    "    TypeError: unhashable type: 'list'\n",
    "    >>>\n",
    "    >>> {\n",
    "    ...     {}: 'dict is not hashable either'\n",
    "    ... }\n",
    "    Traceback (most recent call last):\n",
    "      File \"<stdin>\", line 2, in <module>\n",
    "    TypeError: unhashable type: 'dict'\n",
    "    \n",
    "而用户自定义的对象默认便是可哈希对象，对象哈希值由对象地址计算而来，且任意两个不同对象均不相等：\n",
    "\n",
    "    >>> class A:\n",
    "    ...     pass\n",
    "    ...\n",
    "    >>>\n",
    "    >>> a = A()\n",
    "    >>> b = A()\n",
    "    >>>\n",
    "    >>> hash(a), hash(b)\n",
    "    (-9223372036573452351, -9223372036573452365)\n",
    "    >>>\n",
    "    >>> a == b\n",
    "    False\n",
    "    \n",
    "那么，哈希值如何计算呢？答案是——**哈希函数**。在对象模型部分，我们知道对象行为由类型对象决定。**哈希值**计算作为对象行为中的一种，秘密也隐藏在类型对象中—— tp_hash 函数指针。而内置函数 hash 则依赖类型对象中的 tp_hash 函数，完成哈希值计算并返回。\n",
    "\n",
    "以 str 对象为例，其哈希函数位于 Objects/unicodeobject.c 源文件，unicode_hash 是也：\n",
    "\n",
    "    PyTypeObject PyUnicode_Type = {\n",
    "        PyVarObject_HEAD_INIT(&PyType_Type, 0)\n",
    "        \"str\",              /* tp_name */\n",
    "        sizeof(PyUnicodeObject),        /* tp_size */\n",
    "        // ...\n",
    "\n",
    "        (hashfunc) unicode_hash,        /* tp_hash*/\n",
    "\n",
    "        // ...\n",
    "        unicode_new,            /* tp_new */\n",
    "        PyObject_Del,           /* tp_free */\n",
    "    };\n",
    "    \n",
    "对于用户自定义的对象，可以实现**hash**魔术方法，重写默认哈希值计算方法。举个例子，假设标签类 Tag 的实例对象由 value 字段唯一标识，便可以根据 value 字段实现**哈希函数**以及**相等性**判断：\n",
    "\n",
    "    class Tag:\n",
    "\n",
    "        def __init__(self, value, title):\n",
    "            self.value = value\n",
    "            self.title = title\n",
    "\n",
    "        def __hash__(self):\n",
    "            return hash(self.value)\n",
    "\n",
    "        def __eq__(self, other):\n",
    "            return self.value == other.value\n",
    "            \n",
    "哈希值**使用频率**较高，而且在对象生命周期内均不变。因此，可以在对象内部对哈希值进行缓存，避免重复计算。以 str 对象为例，内部结构中的hash字段便是用于保存哈希值的。\n",
    "\n",
    "理想的哈希函数必须保证哈希值尽量均匀地分布于整个哈希空间，越是相近的值，其哈希值差别应该越大。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 哈希冲突"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "一方面，不同的对象，哈希值有可能相同，另一方面，与哈希值空间相比，哈希表的槽位是非常有限的。因此，存在多个键被映射到哈希索引的同一槽位的可能性，这便是**哈希冲突**！\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_36.png\" width=800 height=600>  \n",
    "\n",
    "解决哈希冲突的常用方法有两种：\n",
    "\n",
    "- 分离链接法 ( separate chaining ) ；\n",
    "- 开放地址法 ( open addressing )；"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "#### 分离链接法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**分离链接法** 为每个哈希槽维护一个链表，所有哈希到同一槽位的键保存到对应的链表中：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_39.png\" width=800 height=600>  \n",
    "\n",
    "如上图，**哈希索引** 每个槽位都接着一个**链表**，初始状态为空；哈希到某个槽位的**键**则保存于对应的链表中。例如，key1 和 key3 都哈希到下标为 3 的槽位，依次保存于槽位对应的链表中。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "#### 开放地址法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Python 采用 开放地址法 ( open addressing )，将数据直接保存于哈希槽位中，如果槽位已被占用，则尝试另一个。\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_40.png\" width=800 height=600>  \n",
    "\n",
    "如上图，key3 哈希到槽位 3 ，但已被 key1 占用了；接着尝试槽位 5 并成功保存。那么，槽位 5 是如何决定的呢？一般而言，第 ii 次尝试在首槽位基础上加上一定的偏移量$d_i$。因此，探测方式因函数$d_i$而异。常见的方法有 线性探测 ( linear probing )以及 平方探测 ( quadratic probing )。\n",
    "\n",
    "**线性探测**，顾名思义， $d_i$是一个线性函数，例如$d_i = 2 * i$: \n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_41.png\" width=800 height=600>  \n",
    "\n",
    "\n",
    "**平方探测**，顾名思义， $d_i$是一个平方函数，例如$d_i = i^2$ ：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_42.png\" width=800 height=600>  \n",
    "\n",
    "**线性探测**和**平方探测**很简单，平方探测似乎更胜一筹。如果哈希表存在局部热点，探测很难快速跳过热点区域，而 **平方探测**则好很多。然而，这两种方法都不够好——因为固定的探测序列加大了冲突的概率。\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_41.png\" width=800 height=600>  \n",
    "\n",
    "如图 key 和 key2 等都哈希到槽位 1 ，由于探测序列式相同的，因此冲突概率很高。Python 对此进行了优化，探测函数参考对象哈希值，生成不同的探测序列，进一步降低哈希冲突的可能性：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_42.png\" width=800 height=600>  \n",
    "\n",
    "Python 探测方法在 lookdict 函数中实现，位于 Objects/dictobject.c 源文件内。关键代码如下：\n",
    "\n",
    "    static Py_ssize_t _Py_HOT_FUNCTION\n",
    "    lookdict(PyDictObject *mp, PyObject *key,\n",
    "             Py_hash_t hash, PyObject **value_addr)\n",
    "    {\n",
    "        size_t i, mask, perturb;\n",
    "        PyDictKeysObject *dk;\n",
    "        PyDictKeyEntry *ep0;\n",
    "\n",
    "    top:\n",
    "        dk = mp->ma_keys;\n",
    "        ep0 = DK_ENTRIES(dk);\n",
    "        mask = DK_MASK(dk);\n",
    "        perturb = hash;\n",
    "        i = (size_t)hash & mask;\n",
    "\n",
    "        for (;;) {\n",
    "            Py_ssize_t ix = dk_get_index(dk, i);\n",
    "            // 省略键比较部分代码\n",
    "\n",
    "            // 计算下个槽位\n",
    "            // 由于参考了对象哈希值，探测序列因哈希值而异\n",
    "            perturb >>= PERTURB_SHIFT;\n",
    "            i = (i*5 + perturb + 1) & mask;\n",
    "        }\n",
    "        Py_UNREACHABLE();\n",
    "    }\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 哈希攻击"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Python 在 3.3 以前，**哈希算法** 只根据对象本身计算哈希值。因此，只要 Python 解释器相同，对象哈希值也肯定相同。我们执行 Python 2 解释器启动一个交互式终端，并计算字符串 fasion 的哈希值：\n",
    "\n",
    "    >>> import os\n",
    "    >>> os.getpid()\n",
    "    2878\n",
    "    >>> hash('fasion')\n",
    "    3629822619130952182\n",
    "    \n",
    "我们再次执行 Python 2 解释器启动另一个交互式终端，发现字符串 fasion 的哈希值保存不变：\n",
    "\n",
    "    >>> import os\n",
    "    >>> os.getpid()\n",
    "    2915\n",
    "    >>> hash('fasion')\n",
    "    3629822619130952182\n",
    "\n",
    "如果一些别有用心的人构造出大量哈希值相同的 key ，并提交给服务器，会发生什么事情呢？例如，向一台 Python 2 Web 服务器 post 一个 json 数据，数据包含大量的 key ，所有 key 的哈希值相同。这意味着哈希表将频繁发生哈希冲突，性能由 **O(1)** 急剧下降为 **O(N)** ，被活生生打垮！这就是 **哈希攻击**。\n",
    "\n",
    "问题很严重，好在应对方法却很简单——为对象加把 **盐** ( salt )。具体做法如下：\n",
    "\n",
    "1. Python 解释器进程启动后，产生一个随机数作为 盐 ；\n",
    "2. 哈希函数同时参考 对象本身 以及 随机数 计算哈希值；\n",
    "\n",
    "这样一来，攻击者无法获悉解释器内部的随机数，也就无法构造出哈希值相同的对象了！Python 自 3.3 以后，哈希函数均采用加盐模式，杜绝了 哈希攻击 的可能性。Python 哈希算法在 Python/pyhash.c 源文件中实现，有兴趣的童鞋可以学习一下，这里就不再展开了。\n",
    "\n",
    "执行 Python 3.7 解释器，启动一个交互式终端，并计算字符串 fasion 的哈希值：\n",
    "\n",
    "    >>> hash('fasion')\n",
    "    7411353060704220518\n",
    "    \n",
    "再次执行 Python 3.7 解释器，启动另一个交互式终端，发现字符串 fasion 的哈希值已经变了：\n",
    "\n",
    "    >>> hash('fasion')\n",
    "    1784735826115825426"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-27T06:27:16.653458Z",
     "start_time": "2021-03-27T06:27:16.638466Z"
    },
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 删除操作"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "现在回过头来讨论 dict 哈希表的 **删除** 操作，以下图这个场景为例：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_45.png\" width=800 height=600>  \n",
    "\n",
    "key1 最先插入，使用了哈希槽位 5 以及存储单元 0 ；紧接着插入 key2 ，使用了哈希槽位 1 以及存储单元 1 ；最后插入 key3 时，由于哈希槽位被 key2 占用，改用槽位 6 。\n",
    "\n",
    "如果需要删除 key2 ，该如何操作呢？假设我们在将哈希槽位设置为 EMPTY ，并将存储单元标记为删除：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_46.png\" width=800 height=600>  \n",
    "\n",
    "这样一来，由于 key3 哈希到的槽位 1 是空的，便误以为 key3 不存在。换句话讲，key3 不翼而飞了！因此，删除元素时，必须将对应的哈希槽设置为一个特殊的标识 DUMMY ，避免中断哈希探测链：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_47.png\" width=800 height=600>  \n",
    "\n",
    "\n",
    "哈希槽位状态常量在 Objects/dict-common.h 头文件中定义：\n",
    "\n",
    "    #define DKIX_EMPTY (-1)\n",
    "    #define DKIX_DUMMY (-2)  /* Used internally */\n",
    "    #define DKIX_ERROR (-3)\n",
    "    \n",
    "那么，被删除的存储单元如何复用呢？Python 压根就没想费这个劲，直接使用新的不就好了吗？假设现在新插入 key4 ，Python 并不理会已删除存储单元 1 ，直接使用新的存储单元 3 ：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_48.png\" width=800 height=600>  \n",
    "\n",
    "是的，存储单元中可能有一些是浪费的，但却无伤大雅。如果存储单元已用完，Python 则执行一次容量调整操作，重新分配一个哈希表，并将所有元素搬过去，简单粗暴：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_49.png\" width=800 height=600>  \n",
    "\n",
    "新哈希表规模由当前 dict 当前元素个数决定，因此容量调整有可能是 **扩容** 、**缩容** 或者 **保持不变** 。无论怎样，新哈希表创建后，便有新存储单元可用了！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## 最佳实践：灵活运用内建容器，提高开发效率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# 映射表\n",
    "city2users = {}\n",
    "\n",
    "# 遍历每位用户\n",
    "for user in users:\n",
    "    # 用户所在城市\n",
    "    city = user.city\n",
    "\n",
    "    # 必要时创建列表对象用于保存该城市下所有用户\n",
    "    if city not in city2users:\n",
    "        city2users[city] = []\n",
    "\n",
    "    # 将用户追加到对应城市的列表中\n",
    "    city2users[city].append(user)\n",
    "    \n",
    "# 借用setdefault方法，我们可以将代码写的更简洁、漂亮；\n",
    "for user in users:\n",
    "    city = user.city\n",
    "    city2users.setdefault(city, []).append(user)  \n",
    "# setdefault 方法先检查给定键是否已存在，未存在则以第二个参数进行初始化，最后返回与键关联的值。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### defaultdict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "标准库 collections 模块中的 defaultdict ，完美解决字典默认值问题。defaultdict 接收一个 default_factory 参数，当访问到不存在的键时，defaultdict 调用 default_factory 为其提供一个初始值。\n",
    "\n",
    "    >>> from collections import defaultdict\n",
    "\n",
    "    # 创建一个 defaultdict 对象\n",
    "    >>> d = defaultdict(list)\n",
    "    >>> d\n",
    "    defaultdict(<class 'list'>, {})\n",
    "\n",
    "    # 访问一个不存在的键不会报错\n",
    "    # defaultdict调用list为其初始化一个空列表\n",
    "    >>> d['foo']\n",
    "    []\n",
    "    >>> d\n",
    "    defaultdict(<class 'list'>, {'foo': []})\n",
    "    \n",
    "这样一来，按城市分组用户的功能可以这么来写：\n",
    "\n",
    "    city2users = defaultdict(list)\n",
    "\n",
    "    for user in users:\n",
    "        city2users[user.city].append(user)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 树结构"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "借助 defaultdict ，我们只需一行代码便可实现一个树形存储容器：\n",
    "\n",
    "    >>> Tree = lambda: defaultdict(Tree)\n",
    "    \n",
    "这行代码的巧妙之处在于 **递归** 。Tree 函数初始化一棵树，树的实际结构是一个 defaultdict 对象。当我们访问一个不存在的树分支时，defaultdict 再次调用 Tree 函数完成子树的初始化！\n",
    "\n",
    "我们接着看看 Tree 如何应用：\n",
    "\n",
    "    # 初始化一棵树\n",
    "    >>> tree = Tree()\n",
    "\n",
    "    # 存入一些数据\n",
    "    >>> tree['fruits']['apple'] = 10\n",
    "    >>> tree['fruits']['pear'] = 20\n",
    "    >>> tree['pets']['cat'] = 3\n",
    "    >>> tree['pets']['dog'] = 1\n",
    "\n",
    "    # 树的第一层有两个分支\n",
    "    >>> tree.keys()\n",
    "    dict_keys(['fruits', 'pets'])\n",
    "\n",
    "    # fruits子树下有两个节点\n",
    "    >>> tree['fruits'].items()\n",
    "    dict_items([('apple', 10), ('pear', 20)])\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-27T07:23:08.271336Z",
     "start_time": "2021-03-27T07:23:08.256345Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "Tree = lambda: defaultdict(Tree)\n",
    "tree = Tree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-27T07:23:44.413259Z",
     "start_time": "2021-03-27T07:23:44.395270Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    ">>> tree['fruits']['apple'] = 10\n",
    ">>> tree['fruits']['pear'] = 20\n",
    ">>> tree['pets']['cat'] = 3\n",
    ">>> tree['pets']['dog'] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-27T07:23:53.748463Z",
     "start_time": "2021-03-27T07:23:53.739482Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    ">>> tree.keys()\n",
    "# dict_keys(['fruits', 'pets']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-27T07:24:26.445572Z",
     "start_time": "2021-03-27T07:24:26.440575Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# fruits子树下有两个节点\n",
    ">>> tree.items()\n",
    "# dict_items([('apple', 10), ('pear', 20)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-27T07:32:44.602908Z",
     "start_time": "2021-03-27T07:32:44.593915Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    ">>> tree['fruits'].items()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Counter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "如果我们只需统计每个城市的用户数，可以使用 int 对象作为 defaultdict 的默认值：\n",
    "\n",
    "    city2total = defaultdict(int)\n",
    "\n",
    "    for user in users:\n",
    "        city2total[user.city] += 1\n",
    "        \n",
    "数量统计是一个非常常见的场景，为此 collections 模块提供了一个更趁手的解决方案—— Counter 类：\n",
    "\n",
    "    from collections import Counter\n",
    "\n",
    "    city2total = Counter()\n",
    "\n",
    "    for user in users:\n",
    "        city2total[user.city] += 1\n",
    "        \n",
    "Counter 还有很多其他高级特性，由于篇幅关系就不再深入讲解了，有兴趣的同学请自行查阅 Python 文档。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## 开发实战：基于最小堆设计任务调度系统"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**堆** ( heap )是一种特殊的树结构，能够快速定位 **最大值** 或 **最小值** ，是实现 **堆排序** 和 **优先队列** 的关键。优先队列主要应用在 **事件处理** 和 **任务调度** 等场景。接下来，我们以 **任务调度** 为例，抛砖引玉。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Python中的堆"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Python 标准库内置了 **优先队列** 实现，这就是 heapq 模块。我们知道堆是一种完全二叉树，可以保存于数组中；而 list 对象就是一种典型的动态数组结构！因此， heapq 将堆维护于 list 对象中，而不是提供一种新容器对象。相反，heapq 提供了几个关键操作函数，可直接操作 list 对象：\n",
    "\n",
    "- heapify ，将 list 对象转化成堆(调整元素顺序以满足堆性质)；\n",
    "- heappush ，将新元素压入堆中；\n",
    "- heappop ，弹出堆顶元素；\n",
    "- etc\n",
    "\n",
    "创建一个列表对象并将其作为一个堆来使用：\n",
    "\n",
    "    heap = []\n",
    "    \n",
    "往堆中压入新元素，被压入元素对象必须 可比较 ，自定义类需要实现 __lt__ 等比较方法：\n",
    "\n",
    "    heappush(heap, item)\n",
    "    \n",
    "heapq 将 list 对象维护成 **最小堆** ，因此 **堆顶** (树的 **根节点** )即为最小值：\n",
    "\n",
    "    smallest = top = heap[0]\n",
    "    \n",
    "当然了，我们也可以将最小值从堆中弹出：\n",
    "\n",
    "    item = heappop(heap)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 古典多线程调度"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "假设我们接到一个需求——设计定时任务执行系统。定时任务由 JobItem 类抽象，executing_ts 是任务执行时间：\n",
    "\n",
    "    class JobItem:\n",
    "\n",
    "        def __init__(self, executing_ts, job):\n",
    "            self.executing_ts = executing_ts\n",
    "            self.job = job\n",
    "            # ...\n",
    "    \n",
    "初学者可能会想到最简单的多线程方案。系统需要同时处理多个定时任务，每个任务由一个线程来执行不就好了吗？这就是古典多线程模型，实例代码如下：\n",
    "\n",
    "import time\n",
    "from threading import Thread\n",
    "\n",
    "    def job_worker(job_item):\n",
    "        # 先等待执行时间到达\n",
    "        time.sleep(job_item.executing_ts - time.time())\n",
    "\n",
    "        # 实际执行任务\n",
    "        process(job_item.job)\n",
    "\n",
    "    def add_job(job_item):\n",
    "        # 启动一个线程来执行任务\n",
    "        Thread(target=job_worker, args=(job_item,)).start()\n",
    "        \n",
    "- job_worker ，工作线程执行函数，线程先睡眠等待执行时间到达( 6 行)，然后调用 process 来执行( 8 行)；\n",
    "- add_job ，添加新定时任务时，启动一个新线程来处理；\n",
    "\n",
    "这个方案虽然很简洁，但也很鸡肋。一方面，创建、销毁线程的开销很大；另一方面，由于线程需要占用不少资源，而系统能够支持的最大线程数相对有限。假设现在有成千上万的定时任务等待执行，系统能撑得住吗？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 调度线程引入"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "采用多线程方案时，需要合理控制工作线程的 **个数** 。我们可以将执行时间已到达的任务放进一个 **就绪任务队列** ，然后启动若干个工作线程来执行就绪任务。新任务执行时间不定，可能有的是一分钟后执行，有的是一天后才执行。那么，问题就转变成——如何判断任务是否就绪？\n",
    "\n",
    "这时，我们可以用另一个线程—— **调度线程** 来完成这个使命。调度线程不断接收新任务，并在任务到期时将其添加至就绪任务队列。如果我们用另一个队列来保存新任务，那么调度线程便是两个队列间的**任务搬运工** ：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_50.png\" width=800 height=600>  \n",
    "\n",
    "- **新任务队列** ，保存新任务，任务创建后即添加到这个队列；\n",
    "\n",
    "- **就绪任务队列** ，保存执行时间已到达的任务；\n",
    "\n",
    "- **调度线程** ，订阅 **新任务队列** ，当任务时间到达时将其添加至 **就绪任务队列(搬运工)**；\n",
    "\n",
    "- **工作线程** ，从 **就绪任务队** 列取出任务并执行(**消费者**)；\n",
    "\n",
    "借助 queue 模块，实现方案中的队列只需两行代码："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from queue import Queue\n",
    "\n",
    "# 新任务队列\n",
    "new_jobs = Queue()\n",
    "\n",
    "# 就绪任务队列\n",
    "ready_jobs = Queue()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "这样一来，添加新任务时，只需将 JobItem 放入 **新任务队列** 即可："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def add_job(job_item):\n",
    "    new_jobs.put(job_item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "工作线程执行逻辑也很简单，一个永久循环便搞定了："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def job_worker():\n",
    "    while True:\n",
    "        # 从就绪队列中取任务\n",
    "        job_item = ready_jobs.get()\n",
    "\n",
    "        # 执行任务\n",
    "        process(job_item.job)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "由于就绪任务一定是所有任务中执行时间最小的，因此可以用一个 **最小堆** 来维护任务集。我们希望任务按执行时间排序，因此需要为 JobItem 编写相关比较方法： "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from functools import total_ordering\n",
    "\n",
    "@total_ordering\n",
    "class JobItem:\n",
    "\n",
    "    def __init__(self, executing_ts, job):\n",
    "        self.executing_ts = executing_ts\n",
    "        self.job = job\n",
    "        # ...\n",
    "\n",
    "    def __eq__(self, other):\n",
    "        return self.executing_ts == other.executing_ts\n",
    "\n",
    "    def __lt__(self, other):\n",
    "        return self.executing_ts < other.executing_ts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "注意到，我们只实现了 __eq__ 和 __lt__ 魔术方法，__gt__ 等其他比较方法均由 total_ordering 装饰器代劳。\n",
    "\n",
    "调度线程只需从新任务队列中取任务并压入最小堆，与此同时检查堆顶任务执行时间是否到达。由于线程需要同时处理两件不同的事情，初学者可能要慌了。不打紧，我们先画一个流程图梳理一下执行逻辑：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_37.png\" width=800 height=600>  \n",
    "\n",
    "线程主体逻辑是一个永久循环，每次循环时：\n",
    "\n",
    "1. 先检查堆顶任务，如果执行时间已到，则移到就绪任务队列并进入下次循环；\n",
    "2. 等待新任务队列，如有新任务到达，则压入堆中并进入下次循环；\n",
    "3. 特别注意，等待新任务时不能永久阻塞，需要根据当前堆顶任务计算等待时间；\n",
    "4. 等待超时便进入下次循环再次检查堆顶任务，因此堆中任务不会被耽搁；\n",
    "\n",
    "理清执行逻辑后，调度线程实现便没有任何难度了，代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from heapq import heappush, heappop\n",
    "\n",
    "def job_dispatcher():\n",
    "    # 任务堆，以执行时间排序的最小堆\n",
    "    heap = []\n",
    "\n",
    "    while True:\n",
    "        # 从新任务队列取任务的等待时间\n",
    "        wait_time = 1\n",
    "\n",
    "        # 取任务前先检查堆中任务\n",
    "        if heap:\n",
    "            # 堆顶任务执行时间最小，也就是最近需要执行的任务\n",
    "            job_item = heap[0]\n",
    "\n",
    "            # 执行时间与当前时间比较\n",
    "            now = time.time()\n",
    "            executing_ts = job_item.executing_ts\n",
    "            if now >= executing_ts:\n",
    "                # 执行时间到达，从堆中弹出并放入就绪队列\n",
    "                heappop(heap)\n",
    "                ready_jobs.put(job_item)\n",
    "\n",
    "                # 继续循环以便再次检查堆顶任务\n",
    "                continue\n",
    "            else:\n",
    "                # 否则计算需要等待的时间\n",
    "                wait_time = executing_ts - now\n",
    "\n",
    "        try:\n",
    "            # 从新任务队列等待并取出新任务\n",
    "            # 请注意，超时时间为当前堆顶任务等待时间\n",
    "            # 超时时间到达后，get停止阻塞，进入下次循环检查并执行堆顶任务\n",
    "            job_item = new_queue.get(timeout=wait_time)\n",
    "\n",
    "            # 将新任务压入堆中\n",
    "            # 注意到，执行时间为元组第一个元素，因此堆以执行时间排序\n",
    "            heappush(heap, job_item)\n",
    "        except:\n",
    "            pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-27T08:03:35.065038Z",
     "start_time": "2021-03-27T08:03:35.051032Z"
    },
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### PriorityQueue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "用过 PriorityQueue 的同学可能会提出这样的解决方案：\n",
    "\n",
    "    import time\n",
    "    from queue import PriorityQueue\n",
    "\n",
    "    jobs = PriorityQueue()\n",
    "\n",
    "    def job_worker():\n",
    "        while True:\n",
    "            # 从优先队列中取任务\n",
    "            job_item = jobs.get()\n",
    "\n",
    "            # 必要时睡眠，以确保任务时间到达才执行\n",
    "            now = time.time()\n",
    "            executing_ts = job_item.executing_ts\n",
    "            if executing_ts > now:\n",
    "                time.sleep(executing_ts - now)\n",
    "\n",
    "            # 执行任务\n",
    "            process(job_item.job)\n",
    "\n",
    "    def add_job(job_item):\n",
    "        jobs.put(job_item)\n",
    "        \n",
    "PriorityQueue 是 Python 标准库提供的优先队列，位于 queue 模块，接口与 Queue 一致。PriorityQueue 底层基于 heapq 实现 get 优先返回 **最小值** 。此外，PriorityQueue 是线程安全的，因此可以在多线程环境中使用：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_52.png\" width=800 height=600>  \n",
    "\n",
    "这个多线程模型很简洁，但这个场景中存在一个 **致命问题** 。\n",
    "\n",
    "假设我们先往队列提交 3 个 1 小时后执行的定时任务，这 3 个任务均被工作线程接收并处理。由于任务时间未到，工作线程先睡眠等待，这似乎问题不大。试想这时再提交一个 1 分钟后执行的定时任务会怎样？\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_53.png\" width=800 height=600>  \n",
    "\n",
    "由于工作线程正在睡眠，新任务需要等到工作线程唤醒，也就是一小时后才有机会被执行！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 优先队列设计思路揭秘"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "当然了，我们可以对 PriorityQueue 进行一定的改造，使得 get 方法阻塞到堆顶任务到期才返回。想要改造 PriorityQueue ，必须知道它的实现方式。只要有源码，一切都好说：\n",
    "\n",
    "    >>> import queue\n",
    "    >>> print(queue.__file__)\n",
    "    /usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/queu\n",
    "    e.py\n",
    "    \n",
    "这里不打算花大量篇幅介绍 PriorityQueue 的源码，鼓励亲们自行研究，必有收获。现在，我们再造一个轮子——从零开始设计自己 PriorityQueue 类，以此领会优先队列的设计思路，并探索新解决方案。\n",
    "\n",
    "首先，我们模仿 Queue 实现优先队列类 PriorityQueue ，将 get 、 put 方法代理到 heappush 和 heappop ："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from heapq import heappush, heappop\n",
    "\n",
    "class PriorityQueue:\n",
    "\n",
    "    def __init__(self):\n",
    "        self.queue = []\n",
    "\n",
    "    def get(self):\n",
    "        if self.queue:\n",
    "            return heappop(self.queue)\n",
    "\n",
    "    def put(self, item):\n",
    "        heappush(self.queue, item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "由于 PriorityQueue 可能同时被多个线程访问，因此必须考虑 线程安全性 。由于 heappush 和 heappop 函数并不是线程安全的，需要给这 self.queue 加上一一个 互斥锁 。因此，我们引入 threading 模块的 Lock 对象："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from  threading import Lock\n",
    "from heapq import heappush, heappop\n",
    "\n",
    "class PriorityQueue:\n",
    "\n",
    "    def __init__(self):\n",
    "        self.queue = []\n",
    "        self.mutex = Lock()\n",
    "\n",
    "    def get(self):\n",
    "        with self.mutex:\n",
    "            if self.queue:\n",
    "                return heappop(self.queue)\n",
    "\n",
    "    def put(self, item):\n",
    "        with self.mutex:\n",
    "            heappush(self.queue, item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "现在多线程导致的 竞争态 已经消除了，但还存在另一个问题：当队列为空时，消费者线程取不到任务。由于消费者并不知道队列何时有新任务，因此只能不断轮询，浪费大量 CPU 时间："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "while True:\n",
    "    result = queue.get()\n",
    "    if result is None:\n",
    "        continue\n",
    "        \n",
    "    # 处理任务"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "如果资源不可用时，消费者线程能够先睡眠，等生产者线程准备好资源后再唤醒就完美了！事实上，操作系统有这样的 **线程同步** 工具，这就是 **条件变量** 。Python 将条件变量封装为 Condition 对象，位于 threading 模块内。借助 Condition 对象，可以轻松实现线程等待和唤醒：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_02\\C2_54.png\" width=800 height=600>  \n",
    "\n",
    "- 条件对象与资源保护锁配合使用；\n",
    "- 资源未准备好，消费者调用 wait 方法等待生产者释放互斥器；\n",
    "- 生产者准备好资源，调用 notify 方法通知消费者；\n",
    "- 消费者被唤醒，wait 方法重新拿到互斥器并返回，消费者再次检查资源状态；\n",
    "\n",
    "引入 Condition 后的代码如下，请结合注释阅读，不再赘述："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "class PriorityQueue:\n",
    "\n",
    "    def __init__(self):\n",
    "        self.queue = []\n",
    "        # 互斥锁\n",
    "        self.mutex = Lock()\n",
    "        # 条件对象\n",
    "        self.more = Condition(self.mutex)\n",
    "\n",
    "    def get(self):\n",
    "        # 先加互斥锁\n",
    "        with self.mutex:\n",
    "            while True:\n",
    "                if not self.queue:\n",
    "                    # 堆为空则等待条件变量\n",
    "                    # wait内部将释放互斥锁，以便生产者线程加锁\n",
    "                    self.more.wait()\n",
    "                    \n",
    "                    # 条件变量满足，线程重新唤醒\n",
    "                    # wait返回前，重新拿到互斥锁\n",
    "                    # 睡眠这段时间，至少有一个生产者压入新数据\n",
    "                    # 新数据有可能被其他线程抢走，因此需要重新检查堆是否为空\n",
    "                    continue\n",
    "\n",
    "                return heappop(self.queue)\n",
    "\n",
    "    def put(self, item):\n",
    "        with self.mutex:\n",
    "            # 将数据压入堆内\n",
    "            heappush(self.queue, item)\n",
    "            # 唤醒等待条件对象的其他线程\n",
    "            self.more.notify()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "至此，我们已经山寨了一个 PriorityQueue ，麻雀虽小五脏俱全！\n",
    "\n",
    "接下来，我们必须改造 PriorityQueue ，保证任务只有到期后才能弹出。因此，get 方法需要检查堆顶任务，未到期则等待。等待期间可能有执行时间更早的任务提交，这时线程必须停止等待重新检查堆顶。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "class PriorityQueue:\n",
    "\n",
    "    def __init__(self):\n",
    "        self.queue = []\n",
    "        # 互斥锁\n",
    "        self.mutex = Lock()\n",
    "        # 条件对象\n",
    "        self.more = Condition(self.mutex)\n",
    "\n",
    "    def get(self):\n",
    "        # 先加互斥锁\n",
    "        with self.mutex:\n",
    "            while True:\n",
    "                if not self.queue:\n",
    "                    # 堆为空则等待条件变量\n",
    "                    # wait内部将释放互斥锁\n",
    "                    self.more.wait()\n",
    "\n",
    "                    # 条件变量满足，线程重新唤醒\n",
    "                    # wait返回前，重新拿到互斥锁\n",
    "                    # 睡眠这段时间，至少有一个生产者压入新任务\n",
    "                    # 新任务有可能被其他线程抢走，因此需要重新检查堆是否为空\n",
    "                    continue\n",
    "\n",
    "                # 检查堆顶任务\n",
    "                job_item = self.queue[0]\n",
    "\n",
    "                # 判断执行时间是否到达\n",
    "                now = time.time()\n",
    "                executing_ts = job_item.executing_ts\n",
    "                if executing_ts > now:\n",
    "                    # 执行时间未到，则继续等待，直到时间达到或者有生产者压入新任务\n",
    "                    self.more.wait(executing_ts - now)\n",
    "                    # 有新任务提交或者等待时间已到，重新检查堆状态\n",
    "                    continue\n",
    "\n",
    "                # 弹出堆顶元素并返回\n",
    "                heappop(self.queue)\n",
    "                return job_item\n",
    "\n",
    "    def put(self, job_item):\n",
    "        with self.mutex:\n",
    "            # 将任务压入堆内，堆以执行时间排序\n",
    "            heappush(self.queue, job_item)\n",
    "            # 唤醒等待条件对象的其他线程\n",
    "            self.more.notify()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "至此，我们完全实现了先前的设想！学习源码后，一路开挂，没有什么是不可能的！\n",
    "\n",
    "最后，谈一谈重复造轮子的问题。\n",
    "\n",
    "在实际工程项目中，最好不要重复造轮子。我们可以站在巨人的肩膀上，调研并选用业内成熟的解决方案，保证项目的 **稳定性** 、 **可维护性** 以及 **开发效率** 。\n",
    "\n",
    "在学习的过程中，鼓励重复造轮子。我们不应该被类库蒙蔽双眼，沦为 API 调用侠。而想要了解一个类库的秘密，最佳途径是模仿着自己实现一个。带着好奇心，保持学习的心态，不断模仿，便总有超越的一天。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# 虚拟机"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "##  Python 程执行过程与字节码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "我们每天都要编写一些 Python 程序，或者用来处理一些文本，或者是做一些系统管理工作。程序写好后，只需敲下 python 命令，便可将程序启动起来并开始执行：\n",
    "\n",
    "    $ python some-program.py\n",
    "    \n",
    "那么，一个文本形式的 .py 文件，是如何一步步转换为能够被 CPU 执行的机器指令的呢？此外，程序执行过程中可能会有 .pyc 文件生成，这些文件又有什么作用呢？带着这些问题我们开始本节的探索。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Python程序执行过程"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "你也许听过这样的说法： Python 是一种解释性语言。这意味着 Python 程序不用编译，只需要用一个解释器来执行。事实真的是这样吗？\n",
    "\n",
    "虽然从行为上看 Python 更像 Shell 脚本这样的解释性语言，但实际上 Python 程序执行原理本质上跟 Java 或者 C# 一样，都可以归纳为 **虚拟机** 和 **字节码** 。 Python 执行程序分为两步：先将程序代码编译成**字节码**，然后启动虚拟机执行字节码：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_01.png\" width=800 height=600>  \n",
    "\n",
    "虽然 python 命令也叫做 Python **解释器** ( Interpreter )，但跟其他脚本语言解释器有本质区别。实际上， Python 解释器包含 **编译器** 以及 **虚拟机** 两部分。当 Python 解释器启动后，主要执行以下两个步骤：\n",
    "\n",
    "1. **编译器** 将 .py 文件中的 Python 源码编译成 **字节码** ；\n",
    "2. **虚拟机** 逐行执行编译器生成的 **字节码** ；\n",
    "\n",
    "因此， .py 文件中的 Python 语句并没有直接转换成机器指令，而是转换成 Python **字节码** 。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### 字节码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "好了，我们知道 Python 程序的 **编译结果** 是字节码，里面应该藏着不少 Python 运行的秘密。因此，不管是为了更深入理解 Python 虚拟机运行机制，还是为了调优 Python 程序运行效率，字节码都是绕不过去的一关。那么， Python 字节码到底长啥样呢？我们如何才能获得一个 Python 程序的字节码呢？\n",
    "\n",
    "为了回答以上问题，我们需要深入 Python 解释器源码，研究 Python **编译器** 。但出于几方面考虑，我不打算深入介绍 Python 编译器：\n",
    "\n",
    "1. Python 编译器工作原理与其他任何语言类似，市面上任何一本编译原理均有介绍；\n",
    "\n",
    "2. 编译原理是计算机基础学科，不是 Python 特有的，不在本专栏的篇幅内；\n",
    "\n",
    "3. 能够影响 Python 编译过程的手段非常有限，研究 Python 编译器对开发工作帮助不大。因此，我们只需要知道 Python 解释器背后有一个编译器负责将源码编译成字节码即可， **字节码以及虚拟机才是我们重点研究的对象** 。\n",
    "\n",
    "那我们还怎么研究字节码呀？别急， Python 提供了一个内置函数 compile 用于即时编译源码。我们只需将待编译源码作为参数调用 compile 函数，即可获得源码的编译结果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 源码编译"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "接下来，我们调用 compile 函数编译一个例子程序，以此演示该函数的用法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-27T09:08:07.710158Z",
     "start_time": "2021-03-27T09:08:07.693155Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "PI = 3.14\n",
    "\n",
    "def circle_area(r):\n",
    "    return PI * r ** 2\n",
    "\n",
    "class Dog(object):\n",
    "  \n",
    "    def __init__(self, name):\n",
    "            self.name = name\n",
    "\n",
    "    def yelp(self):\n",
    "            print('woof, i am', self.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "假设这段源码保存于 demo.py 文件，开始编译之前需要将源码从文件中读取出来：\n",
    "\n",
    "    >>> text = open('demo.py').read()\n",
    "    >>> print(text)\n",
    "    PI = 3.14\n",
    "\n",
    "    def circle_area(r):\n",
    "        return PI * r ** 2\n",
    "\n",
    "    class Dog(object):\n",
    "\n",
    "        def __init__(self, name):\n",
    "            self.name = name\n",
    "\n",
    "        def yelp(self):\n",
    "            print('woof, i am', self.name)\n",
    "            \n",
    "接着，调用 compile 函数编译源码：\n",
    "\n",
    "    >>> result = compile(text, 'demo.py', 'exec')\n",
    "    \n",
    "compile 函数必填的参数有 3 个：\n",
    "\n",
    "- source ，待编译 源码 ；\n",
    "- filename ，源码所在 文件名 ；\n",
    "- mode ， 编译模式 ， exec 表示将源码当做一个模块来编译；\n",
    "\n",
    "顺便提一下， compile 函数有 3 中不同的 编译模式 可供选择：\n",
    "\n",
    "- exec ，用于编译模块源码；\n",
    "- single ，用于编译一个单独的 Python 语句(交互式下)；\n",
    "- eval ，用于编译一个 eval 表达式；\n",
    "\n",
    "compile 详细用法请参考 Python 文档，运行 help 内建函数可快速查看：\n",
    "\n",
    "    >>> help(compile)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### PyCodeObject"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "我们接着看源码编译结果到底是个什么东西：\n",
    "\n",
    "    >>> result\n",
    "    <code object <module> at 0x103d21150, file \"demo.py\", line 1>\n",
    "    >>> result.__class__\n",
    "    <class 'code'\n",
    "    \n",
    "看上去我们得到了一个 代码对象 ，代码对象有什么特别的呢？接着顺势扒开 Include/code.h 看一看，我们找到了代表代码对象的 C 结构体 PyCodeObject 。 PyCodeObject 定义如下：\n",
    "\n",
    "    /* Bytecode object */\n",
    "    typedef struct {\n",
    "        PyObject_HEAD\n",
    "        int co_argcount;            /* #arguments, except *args */\n",
    "        int co_kwonlyargcount;      /* #keyword only arguments */\n",
    "        int co_nlocals;             /* #local variables */\n",
    "        int co_stacksize;           /* #entries needed for evaluation stack */\n",
    "        int co_flags;               /* CO_..., see below */\n",
    "        int co_firstlineno;         /* first source line number */\n",
    "        PyObject *co_code;          /* instruction opcodes */\n",
    "        PyObject *co_consts;        /* list (constants used) */\n",
    "        PyObject *co_names;         /* list of strings (names used) */\n",
    "        PyObject *co_varnames;      /* tuple of strings (local variable names) */\n",
    "        PyObject *co_freevars;      /* tuple of strings (free variable names) */\n",
    "        PyObject *co_cellvars;      /* tuple of strings (cell variable names) */\n",
    "        /* The rest aren't used in either hash or comparisons, except for co_name,\n",
    "           used in both. This is done to preserve the name and line number\n",
    "           for tracebacks and debuggers; otherwise, constant de-duplication\n",
    "           would collapse identical functions/lambdas defined on different lines.\n",
    "        */\n",
    "        Py_ssize_t *co_cell2arg;    /* Maps cell vars which are arguments. */\n",
    "        PyObject *co_filename;      /* unicode (where it was loaded from) */\n",
    "        PyObject *co_name;          /* unicode (name, for reference) */\n",
    "        PyObject *co_lnotab;        /* string (encoding addr<->lineno mapping) See\n",
    "                                       Objects/lnotab_notes.txt for details. */\n",
    "        void *co_zombieframe;       /* for optimization only (see frameobject.c) */\n",
    "        PyObject *co_weakreflist;   /* to support weakrefs to code objects */\n",
    "        /* Scratch space for extra data relating to the code object.\n",
    "           Type is a void* to keep the format private in codeobject.c to force\n",
    "           people to go through the proper APIs. */\n",
    "        void *co_extra;\n",
    "    } PyCodeObject;\n",
    "    \n",
    "代码对象 PyCodeObject 用于存储编译结果，包括字节码 以及代码涉及的 常量 名字 等等。关键字段包括：\n",
    "\n",
    "|字段|用途|\n",
    "|:-:|:-:|\n",
    "|co_argcount|\t参数个数|\n",
    "|co_kwonlyargcount|\t关键字参数个数|\n",
    "|co_nlocals|\t局部变量个数|\n",
    "|co_stacksize|\t执行代码所需栈空间|\n",
    "|co_flags|\t标识|\n",
    "|co_firstlineno|\t代码块首行行号|\n",
    "|co_code|\t指令操作码，也就是字节码|\n",
    "|co_consts|\t常量列表|\n",
    "|co_names|\t名字列表|\n",
    "|co_varnames|局部变量名列表|\n",
    "|co_freevars||\t\n",
    "|co_cellvars||\t\n",
    "\n",
    "我们终于得到了字节码，尽管它现在看上去如同天书一般：\n",
    "\n",
    "    >>> result.co_code\n",
    "    b'd\\x00Z\\x00d\\x01d\\x02\\x84\\x00Z\\x01G\\x00d\\x03d\\x04\\x84\\x00d\\x04e\\x02\\x83\\x03Z\\x03d\\x05S\\x00'\n",
    "    \n",
    "字节码我们现在还无法读懂，放一放。接着研究其他字段，看看名字列表，包含代码对象涉及的所有名字：\n",
    "\n",
    "    >>> result.co_names\n",
    "    ('PI', 'circle_area', 'object', 'Dog')\n",
    "    \n",
    "常量列表则包括代码对象涉及的所有常量：\n",
    "\n",
    "    >>> result.co_consts\n",
    "    (3.14, <code object circle_area at 0x10356c5d0, file \"demo.py\", line 3>, 'circle_area', <code object Dog at 0x10356cae0, file \"demo.py\", line 6>, 'Dog', None)\n",
    "    \n",
    "常量列表里还藏着两个代码对象！其中一个对应着 circle_area 函数体，另一个对应着 Dog 类定义体。回想起 Python **作用域** 的划分方式，很自然地联想到： **每个作用域对应着一个代码对象** ！如果这个假设成立， Dog 代码对象的常量列表应该还藏着两个代码对象，分别代表 **init** 方法和 yelp 方法的函数体：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_02.png\" width=800 height=600>  \n",
    "\n",
    "进一步研究代表类 Dog 的代码对象，我们发现事实确实如此：\n",
    "\n",
    "    >>> dog_code = result.co_consts[3]\n",
    "    >>> dog_code\n",
    "    <code object Dog at 0x10356cae0, file \"demo.py\", line 6>\n",
    "    >>> dog_code.co_consts\n",
    "    ('Dog', <code object __init__ at 0x10356c420, file \"demo.py\", line 8>, 'Dog.__init__', <code object yelp at 0x10356c930, file \"demo.py\", line 11>, 'Dog.yelp', None)\n",
    "    \n",
    "因此，我们得到以下结论： Python 源码编译后，每个作用域都对应着一个代码对象，子作用域代码对象位于父作用域代码对象的常量列表里，层级一一对应。\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_03.png\" width=800 height=600>  \n",
    "\n",
    "至此，我们对 Python 源码的编译结果—— **代码对象** 以及其中的 **字节码** 有了最基本的认识。虽然代码对象中的很多字段我们还没来得及研究，但不要紧，我们将在 **虚拟机** 、 **函数机制** 、 **类机制** 的学习中一一揭开这些秘密。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 反编译"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "字节码是一堆不可读的字节序列，跟二进制机器码一样。我们想读懂机器码，可以将其反汇编。那么，字节码是不是也可以反编译呢？答案是肯定的—— dis 模块就是干这个事的：\n",
    "\n",
    "    >>> import dis\n",
    "    >>> dis.dis(result.co_code)\n",
    "              0 LOAD_CONST               0 (0)\n",
    "              2 STORE_NAME               0 (0)\n",
    "              4 LOAD_CONST               1 (1)\n",
    "              6 LOAD_CONST               2 (2)\n",
    "              8 MAKE_FUNCTION            0\n",
    "             10 STORE_NAME               1 (1)\n",
    "             12 LOAD_BUILD_CLASS\n",
    "             14 LOAD_CONST               3 (3)\n",
    "             16 LOAD_CONST               4 (4)\n",
    "             18 MAKE_FUNCTION            0\n",
    "             20 LOAD_CONST               4 (4)\n",
    "             22 LOAD_NAME                2 (2)\n",
    "             24 CALL_FUNCTION            3\n",
    "             26 STORE_NAME               3 (3)\n",
    "             28 LOAD_CONST               5 (5)\n",
    "             30 RETURN_VALUE\n",
    "             \n",
    " 看到没，字节码反编译后的结果多么像汇编语言！其中，第一列是字节码 **偏移量** ，第二列是 **指令** ，第三列是 **操作数** 。以第一条字节码为例， LOAD_CONST 指令将常量加载进栈，常量下标由操作数给出。而下标为 0 的常量是：\n",
    " \n",
    "     >>> result.co_consts[0]\n",
    "    3.14\n",
    " \n",
    "我们成功解开了第一条字节码：将常量 3.14 加载到栈！对其他字节码的解读也是类似的。\n",
    "\n",
    "由于代码对象保存了常量、名字等上下文信息，因此直接对代码对象进行反编译可以得到更为清晰的结果：\n",
    "\n",
    "    >>> dis.dis(result)\n",
    "      1           0 LOAD_CONST               0 (3.14)\n",
    "                  2 STORE_NAME               0 (PI)\n",
    "\n",
    "      3           4 LOAD_CONST               1 (<code object circle_area at 0x10356c5d0, file \"demo.py\", line 3>)\n",
    "                  6 LOAD_CONST               2 ('circle_area')\n",
    "                  8 MAKE_FUNCTION            0\n",
    "                 10 STORE_NAME               1 (circle_area)\n",
    "\n",
    "      6          12 LOAD_BUILD_CLASS\n",
    "                 14 LOAD_CONST               3 (<code object Dog at 0x10356cae0, file \"demo.py\", line 6>)\n",
    "                 16 LOAD_CONST               4 ('Dog')\n",
    "                 18 MAKE_FUNCTION            0\n",
    "                 20 LOAD_CONST               4 ('Dog')\n",
    "                 22 LOAD_NAME                2 (object)\n",
    "                 24 CALL_FUNCTION            3\n",
    "                 26 STORE_NAME               3 (Dog)\n",
    "                 28 LOAD_CONST               5 (None)\n",
    "                 30 RETURN_VALUE\n",
    "                 \n",
    " 注意到，操作数指定的常量或名字的实际值在旁边的括号内列出。另外，字节码以语句为单位进行分组，中间以空行隔开，语句行号在字节码前面给出。 PI = 3.14 这个语句编译成以下两条字节码：\n",
    " \n",
    "      1           0 LOAD_CONST               0 (3.14)\n",
    "                  2 STORE_NAME               0 (PI)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### pyc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "如果将 demo 作为模块导入， Python 将在 demo.py 文件所在目录下生成 .pyc 文件：\n",
    "\n",
    "    >>> import demo\n",
    "    \n",
    "\n",
    "    $ ls __pycache__ \n",
    "    demo.cpython-37.pyc\n",
    "    \n",
    "pyc 文件保存经过序列化处理的代码对象 PyCodeObject 。这样一来， Python 后续导入 demo 模块时，直接读取 pyc 文件并反序列化即可得到代码对象，避免了重复编译导致的开销。只有 demo.py 有新修改(时间戳比 pyc 文件新)， Python 才会重新编译。\n",
    "\n",
    "因此， Python 中的 .py 文件可以类比 Java 中的 .java 文件，都是源码文件；而 .pyc 文件可以类比 .class 文件，都是编译结果(字节码)。只不过 Java 程序需要先用编译器 javac 命令来编译，再用虚拟机 java 命令来执行；而 Python 解释器把这个两个活都干了，更加智能。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 总结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Python **程序** 由 **解释器** python 命令执行， Python 解释器中包含一个 **编译器** 和一个 **虚拟机** 。 Python 解释器执行 Python 程序时，分为以下两步：\n",
    "\n",
    "1. **编译器** 将 .py 文件中的 Python 源码编译成 **字节码** ；\n",
    "2. **虚拟机** 逐行执行编译器生成的 **字节码** ；\n",
    "\n",
    "Python 源码的编译结果是代码对象 PyCodeObject ，对象中保存了 **字节码** 、 **常量** 以及 **名字** 等信息，代码对象与源码作用域一一对应。 Python 将编译生成的代码对象保存在 .pyc 文件中，以避免不必要的重复编译，提高效率。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## 面试必问概念精讲：作用域与名字空间"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "作为 Python 面试官，每次面试中我几乎都会和候选人聊起 作用域 以及 名字空间 等基本概念。但就算这么基础的内容，也有不少人没有完全掌握，也因此与工作机会失之交臂。\n",
    "\n",
    "    PI = 3.14\n",
    "\n",
    "    def circle_area(r):\n",
    "        return PI * r ** 2\n",
    "\n",
    "    class Dog(object):\n",
    "\n",
    "        def __init__(self, name):\n",
    "            self.name = name\n",
    "\n",
    "        def yelp(self):\n",
    "            print('woof, i am', self.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 名字绑定"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### 赋值"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "在 Python 中，变量只是一个与实际对象绑定起来的名字，变量定义本质上就是建立名字与对象的约束关系。因此，赋值语句本质上就是建立这样的约束关系，将右边的对象与左边的名字绑定在一起：\n",
    "\n",
    "    a = 1\n",
    "    \n",
    "我经常在面试中问：除了赋值语句，还有哪些语句可以完成名字绑定？能准确回答的候选人寥寥无几。实际上，除了赋值语句外， Python 中还有好几类语句均与名字绑定相关，我们接着一一介绍。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### 模块导入"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "我们导入模块时，也会在当前上下文创建一个名字，并与被导入对象绑定：\n",
    "\n",
    "    import xxx\n",
    "    from xxx import yyy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### 函数、类定义"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "我们定义函数 / 类时，本质上是创建了一个函数 / 类对象，然后将其与函数 / 类名绑定：\n",
    "\n",
    "    def circle_area(r):\n",
    "        return PI * r ** 2\n",
    "\n",
    "    class Dog(object):\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### as 关键字"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "除此此外， as 关键字也可以在当前上下文建立名字约束关系：\n",
    "\n",
    "    import xxx as yyy\n",
    "    from xxx import yyy as zzz\n",
    "\n",
    "    with open('/some/file') as f:\n",
    "        pass\n",
    "\n",
    "    try:\n",
    "        # do something\n",
    "    except SomeError as e:\n",
    "        # handle error\n",
    "        \n",
    "以上这几类语句均可在当前上下文建立名字约束，有着与赋值语句类似的行为，因此可以看作是 广义的赋值语句 。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 作用域"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "现在问题来了，一个名字引入后，它的可见范围有多大呢？\n",
    "\n",
    "我们以一个面试真题开始讨论：以下例子中 3 个 print 语句分别输出什么？\n",
    "\n",
    "    a = 1\n",
    "\n",
    "    def f1():\n",
    "        print(a)\n",
    "\n",
    "    def f2():\n",
    "        a = 2\n",
    "        print(a)\n",
    "\n",
    "    print(a)\n",
    "    \n",
    "例子中，第 1 行引入的名字 a 对整个模块都可见，第 4 行和第 10 行均可访问到它，因此这两个地方输出 1 ；而第 7 行引入的名字 a 却只有函数 f2 内部可以访问到，第 8 行优先访问内部定义的 a ，因此这里将输出 2 。\n",
    "\n",
    "由此可见，在不同的代码区域引入的名字，其影响范围是不一样的。第 1 行定义的 a 可以影响到 f1 ，而 f2 中定义的 a 却不能。再者，一个名字可能在多个代码区域中定义，但最终只能使用其中一个。\n",
    "\n",
    "一个名字能够施加影响的程序正文区域，便是该名字的 **作用域** 。在 Python 中，一个名字在程序中某个区域能否起作用，是由名字引入的位置决定的，而不是运行时动态决定的。因此， Python 具有 **静态作用域** ，也称为 **词法作用域** 。那么，程序的作用域是如何划分的呢？\n",
    "\n",
    "Python 在编译时，根据语法规则将代码划分为不同的 **代码块** ，每个代码块形成一个 **作用域** 。首先，整个 .py 文件构成最顶层的作用域，这就是 **全局作用域** ，也称为 **模块作用域** ；其次，当代码遇到 **函数定义** ，函数体成为当前作用域的 **子作用域** ；再次，当代码遇到 **类定义** ，类定义体成为当前作用域的子作用域。\n",
    "\n",
    "一个名字在某个作用域引入后，它的影响范围就被限制在该作用域内。其中，全局作用域对所有直接或间接内嵌于其中的子作用域可见；函数作用域对其直接子作用域可见，并且可以传递。\n",
    "\n",
    "按照这个划分方式，真题中的代码总共有 3 个作用域： A 为最外层作用域，即全局作用域； f1 函数体形成作用域 B ，是 A 的子作用域； f2 函数体又形成作用域 C ，也是 A 的子作用域。\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_04.png\" width=800 height=600>  \n",
    "\n",
    "作用域 A 定义的变量 a 对于对 A 及其子作用域 B 、 C 可见，因此 f1 也可以访问到。理论上， f2 也可以访问到 A 中的 a ，只不过其作用域 C 也定义了一个 a ，优先访问本作用域内的。 C 作用域内定义的任何名字，对 A 和 B 均不可见。\n",
    "\n",
    "A B C 三个作用域嵌套关系如左下所示，访问关系如右下所示：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_05.png\" width=800 height=600>  \n",
    "\n",
    "箭头表示访问关系，例如作用域 B 中的语句可以访问到作用域 A 中的名字，反过来则不行。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### 闭包作用域"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "这个例子借助闭包实现提示信息定制功能：\n",
    "\n",
    "    pi = 3.14\n",
    "\n",
    "    def circle_area_printer(hint):\n",
    "\n",
    "        def print_circle_area(r):\n",
    "            print(hint, pi * r ** 2)\n",
    "\n",
    "        return print_circle_area\n",
    "\n",
    "    circle_area_en = circle_area_printer('Circle Area:')\n",
    "    circle_area_zh = circle_area_printer('圆面积：')\n",
    "\n",
    "    circle_area_en(2)\n",
    "    circle_area_zh(3)\n",
    "\n",
    "根据前面介绍的规则，我们对代码进行作用域划分，结果如下：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_06.png\" width=800 height=600>  \n",
    "\n",
    "A B C 三个作用域嵌套关系如左下所示，访问关系如右下所示：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_07.png\" width=800 height=600>  \n",
    "\n",
    "毫无疑问， B C 均在全局作用域 A 内，因此都可以访问到 A 中的名字。由于 B 是函数作用域，对其子作用域 C 可见。因此， hint 属于 B 作用域，而位于 C 作用域的语句可以访问它，也就不奇怪了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### 类作用域"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "我们接着以一个简单的类为例，考察类作用域：\n",
    "\n",
    "    slogan = 'life is short, use python.'\n",
    "\n",
    "    class Dog(object):\n",
    "\n",
    "        group = ''\n",
    "\n",
    "        def __init__(self, name):\n",
    "            self.name = name\n",
    "\n",
    "        def yelp(self):\n",
    "            print('woof,', slogan)\n",
    "\n",
    "        def yelp_name(self):\n",
    "            print('woof, i am', self.name)\n",
    "\n",
    "        def yelp_group(self):\n",
    "            print('woof, my group is', group)\n",
    "            \n",
    "根据前面介绍的规则，我们对代码进行作用域划分，结果如下：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_08.png\" width=800 height=600>  \n",
    "\n",
    "其中， B 对应着类定义体，暂且叫做类作用域。各个作用域嵌套关系以及访问关系分别如下：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_09.png\" width=800 height=600>  \n",
    "\n",
    "同样，全局作用域 A 对其他所有内嵌于其中的作用域可见。因此，函数 yelp 作用域 D 可以访问到全局作用域 A 中的名字 slogan 。但是由于 B 不是函数作用域，对其子作用域不可见。因此， yelp_group 函数作用域 F 访问不到类作用域 B 中的名字 group ，而 group 在全局作用域 A 中未定义，第 17 行便抛异常了。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### 复杂嵌套"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "##### 函数 - 类\n",
    "\n",
    "在 Python 中，类可以动态创建，甚至在函数中返回。在函数中创建并返回类，可以按函数参数对类进行动态定制，有时很有用。那么，这种场景中的作用域又该如何划分呢？我们一起来看一个简单的例子：\n",
    "\n",
    "    slogan = 'life is short, use python.'\n",
    "\n",
    "    def make_dog(group_name):\n",
    "\n",
    "        class Dog(object):\n",
    "\n",
    "            group = group_name\n",
    "\n",
    "            def __init__(self, name):\n",
    "                self.name = name\n",
    "\n",
    "            def yelp(self):\n",
    "                print('woof,', slogan)\n",
    "\n",
    "            def yelp_name(self):\n",
    "                print('woof, i am', self.name)\n",
    "\n",
    "            def yelp_group(self):\n",
    "                print('woof, my group is', self.group)\n",
    "\n",
    "        return Dog\n",
    "\n",
    "    if __name__ == '__main__':\n",
    "        Dog = make_dog('silly-dogs')\n",
    "\n",
    "        tom = Dog(name='tom')\n",
    "        tom.yelp_group()\n",
    "\n",
    "这个例子借助函数实现类属性 group 动态定制，以不同的 group_name 调用函数即可获得不同的 Dog 类。根据前面介绍的规则，我们对代码进行作用域划分，结果如下：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_10.png\" width=800 height=600>  \n",
    "\n",
    "各个作用域嵌套关系以及访问关系分别如下：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_11.png\" width=800 height=600>  \n",
    "\n",
    "同样，全局作用域 A 对其他所有内嵌于其中的作用域可见。由于作用域 B 是函数作用域，因此子作用域 C 中的语句能够范围 B 中的名字。\n",
    "\n",
    "经过以上分析，例子程序应该输出 woof, my group is silly-dogs 。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "##### 类 - 类\n",
    "\n",
    "我们接着考察类嵌套的情形：\n",
    "\n",
    "    class Foo(object):\n",
    "\n",
    "        bar_name = 'BAR'\n",
    "\n",
    "        class Bar(object):\n",
    "\n",
    "            name = bar_name\n",
    "\n",
    "    if __name__ == '__main__':\n",
    "        bar = Foo.Bar()\n",
    "        print(bar.name)\n",
    "        \n",
    "这个例子没有实际含义，纯粹为了考察嵌套类作用域问题。根据前面介绍的规则，我们对代码进行作用域划分：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_12.png\" width=800 height=600>  \n",
    "\n",
    "各个作用域嵌套关系以及访问关系分别如下：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_13.png\" width=800 height=600>  \n",
    "\n",
    "这里需要注意的是，类作用域 B 对子作用域 C 不可见，因此第 7 行就抛异常了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 名字空间"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**作用域** 是语法层面的概念，是静态的。当程序开始执行后，作用域中的名字绑定关系需要存储在某个地方，这个地方就是 **名字空间** 。由于名字绑定关系是有 **名字** 和 **对象** 组成的键值对，因而 dict 对象是理想的存储容器。\n",
    "\n",
    "接下来，我们以计算圆面积的例子程序接着考察作用域背后的运行时实体 —— **名字空间** 。\n",
    "    # circle.py\n",
    "\n",
    "    pi = 3.14\n",
    "\n",
    "    def circle_area_printer(hint):\n",
    "\n",
    "        def print_circle_area(r):\n",
    "            print(hint, pi * r ** 2)\n",
    "\n",
    "        return print_circle_area"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "##### Globals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "在 Python 中，每个 **模块** 背后都有一个 dict 对象，用于存储 **全局作用域** 中的名字，这就是 **全局名字空间** ( Globals )。在上面这个例子中，全局名字空间至少包含两个名字： pi 和 circle_area_printer 。由此可见， Python 的全局名字空间是以 **模块** 为单位划分的，而不是全局统一的。\n",
    "\n",
    "其他模块如果也需要使用 pi ，需要借助 import 语句将其导入。模块导入后，我们得到一个模块对象 (假设例子代码位于 circle.py )：\n",
    "\n",
    "    >>> import circle\n",
    "    >>> type(circle)\n",
    "    <class 'module'>\n",
    "    \n",
    "接着，我们通过模块对象属性查找，便可得到 circle 模块全局名字中间中的 pi ：\n",
    "\n",
    "    >>> print(circle.pi)\n",
    "    3.14\n",
    "    \n",
    "此外，我们还可以进一步确认 _circle_area printer 函数也可以通过模块对象属性的方式来访问：\n",
    "\n",
    "    >>> dir(circle)\n",
    "    ['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'circle_area_printer', 'pi']\n",
    "    \n",
    "在 Python 中，一个对象可以访问哪些属性，称为对象的 **属性空间** 。由于属性也是键值对，因此一般也是用 dict 来存储。通过观察以上代码行为，我们得到一个结论：模块的 **属性空间** 以及 **全局名字空间** 是同一个东西，都藏身于同一个 dict 对象。那么，我们怎么找到这个特殊的 dict 对象呢？答案是：\n",
    "\n",
    "    circle.__dict__\n",
    "    \n",
    "全局名字空间藏身于模块对象背后的 dict 对象中：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_14.png\" width=800 height=600>  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "##### Locals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Python 执行一个作用域内的代码时，需要一个容器来存储当前作用域内的名字，这就是 **局部名字空间** ( Locals )。\n",
    "\n",
    "当 Python 执行函数 circle_area_printer 时，将分配一个栈帧对象保存上下文信息以及执行状态，这个栈帧对象就是后面章节要介绍的 PyFrameObject 。作为代码执行时必不可少上下文信息之一，全局名字空间和局部名字空间也在栈帧对象上记录：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_15.png\" width=800 height=600>  \n",
    "\n",
    "代码语句中涉及的名字查找，均在这个两个名字空间中进行：先查找局部名字空间，再查找全局名字空间。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "##### Enclosings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "在作用域存在嵌套的情况下， Python 将内层代码块中依赖的所有外层名字存储在一个容器内，这就是 **闭包名字空间** ( Enclosings )。\n",
    "\n",
    "当 Python 执行函数 print_circle_area 时，依赖上层作用域中的名字 hint ， hint 保存与一个独立的名字空间中：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_16.png\" width=800 height=600>  \n",
    "\n",
    "当 Python 执行到语句 print(hint, pi * r ** 2) ，按照 Local Enclosing Global 这样的顺序查找语句中涉及的名字。其中，名字 hint 在 Enclosing 中找到，；名字 pi 在 Global 中找到；名字 r 在 Local 中找到。\n",
    "\n",
    "那么， print 函数又是如何找到的呢？这就要说到 **内建名字空间** 。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "##### Builtin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Python 在 builtin 模块中提供了很多内建函数和类型，构成运行时的另一个名字空间 **内建名字空间** ( Builtin )。像 print 这样的内建函数或类型，均需要在这个名字空间中查找：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_17.png\" width=800 height=600>  \n",
    "\n",
    "顺便提一下，全局名字空间中有一个名字指向内建名字空间：\n",
    "\n",
    "    >>> import builtins\n",
    "    >>> circle.__builtins__ is builtins.__dict__\n",
    "    True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 属性空间"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Python 是一个动态语言，在运行时可以很灵活地为一些对象设置新属性。例如：\n",
    "\n",
    "    >>> class A(object):\n",
    "    ...     pass\n",
    "    ... \n",
    "    >>> a = A()\n",
    "    >>> a.value = 'abc'\n",
    "    >>> a.value\n",
    "    'abc'\n",
    "    \n",
    "同样，对象 a 的属性在 Python 内部也是存储在 dict 对象中的，这就是该对象的 属性空间 ：\n",
    "\n",
    "    >>> a.__dict__\n",
    "    {'value': 'abc'}\n",
    "    \n",
    "修改代表对象属性空间的 dict 对象，将影响属性查找结果。由于属性空间中没有名字 name ，因此属性查找失败：\n",
    "\n",
    "    # 由于属性空间中没有名字name，因此属性查找失败\n",
    "    >>> a.name\n",
    "    Traceback (most recent call last):\n",
    "      File \"<stdin>\", line 1, in <module>\n",
    "    AttributeError: 'A' object has no attribute 'name'\n",
    "    \n",
    "当我们往 dict 对象中添加名字 name 后，属性查找便成功了：\n",
    "\n",
    "     >>> a.__dict__['name'] = 'tom'\n",
    "    >>> a.name\n",
    "    'tom'   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 总结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "在 Python 中，一个名字 (变量) 可见范围由 **作用域** 决定，而程序作用域由语法静态划分，划分规则提炼如下：\n",
    "\n",
    "- .py 文件 (模块) 最外层为 **全局作用域** ；\n",
    "- 遇到函数定义，函数体形成子作用域；\n",
    "- 遇到类定义，类定义体形成子作用域；\n",
    "- 名字仅在其作用域以内可见；\n",
    "- 全局作用域对其他所有作用域可见；\n",
    "- 函数作用域对其直接子作用域可见，并且可以传递 ( **闭包** )；\n",
    "\n",
    "与 **作用域** 相对应， Python 在运行时借助 dict 对象保存作用域中的名字，构成动态的 **名字空间** 。这样的名字空间总共有 4 个：\n",
    "\n",
    "- 内建名字空间\n",
    "- 全局名字空间\n",
    "- 闭包名字空间\n",
    "- 局部名字空间\n",
    "Python 语句在查找名字时，按照 Local Enclosing Global Builtin 这样的顺序在 4 个名字空间中查找，这也就是所谓的 **LEGB** 规则。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## 原来虚拟机是一颗软件 CPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "上一小节，我们研究了 Python 程序的编译过程，以及编译产物—— **代码对象** 。 **编译器** 依照语法规则对源码进行 **作用域** 划分，并以此为单位编译源码，最终为每个作用域生成一个代码对象。代码对象则保存了 **字节码** ，以及相关 **名字** 、 **常量** 等静态上下文信息。\n",
    "\n",
    "**编译器** 将源码 **编译** 成代码对象后，便将接力棒传给 **虚拟机** ，由虚拟机负责 **执行** 。那么， Python 虚拟机如何解析并执行字节码指令呢？与语法作用域相对应的运行时 **名字空间** ，在虚拟机中又是如何动态维护的呢？带着这些疑问，我们开始本节关于 **虚拟机** 以及 **字节码** 执行过程的探索。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### PyFrameObject"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "由于代码对象是静态的， Python 虚拟机在执行代码对象时，需要由一个辅助对象来维护执行上下文。那么，这个执行上下文需要包含什么信息呢？我们先根据已经掌握的知识大开脑洞猜测一下：\n",
    "\n",
    "首先，我们需要一个动态容器，来存储代码对象作用域中的名字，这也就是 局部名字空间 ( Locals )。同理，上下文信息还需要记录 **全局名字空间** ( Globals )以及 **内建名字空间** ( Builtins )的具体位置，确保相关名字查找顺畅。\n",
    "\n",
    "其次，虚拟机需要保存当前执行字节码指令的编号，就像 CPU 需要一个寄存器( IP )保存当前执行指令位置一样。\n",
    "\n",
    "因此，执行上下文理论上至少要包括以下这两方面信息：\n",
    "\n",
    "- 名字空间\n",
    "- 当前字节码位置\n",
    "\n",
    "接下来，我们请出执行上下文的真身—— **栈帧对象** PyFrameObject ，看看我们有没有猜对。 PyFrameObject 在头文件 Include/frameobject.h 中定义：\n",
    "\n",
    "    typedef struct _frame {\n",
    "        PyObject_VAR_HEAD\n",
    "        struct _frame *f_back;      /* previous frame, or NULL */\n",
    "        PyCodeObject *f_code;       /* code segment */\n",
    "        PyObject *f_builtins;       /* builtin symbol table (PyDictObject) */\n",
    "        PyObject *f_globals;        /* global symbol table (PyDictObject) */\n",
    "        PyObject *f_locals;         /* local symbol table (any mapping) */\n",
    "        PyObject **f_valuestack;    /* points after the last local */\n",
    "        /* Next free slot in f_valuestack.  Frame creation sets to f_valuestack.\n",
    "           Frame evaluation usually NULLs it, but a frame that yields sets it\n",
    "           to the current stack top. */\n",
    "        PyObject **f_stacktop;\n",
    "        PyObject *f_trace;          /* Trace function */\n",
    "        char f_trace_lines;         /* Emit per-line trace events? */\n",
    "        char f_trace_opcodes;       /* Emit per-opcode trace events? */\n",
    "\n",
    "        /* Borrowed reference to a generator, or NULL */\n",
    "        PyObject *f_gen;\n",
    "\n",
    "        int f_lasti;                /* Last instruction if called */\n",
    "        /* Call PyFrame_GetLineNumber() instead of reading this field\n",
    "           directly.  As of 2.3 f_lineno is only valid when tracing is\n",
    "           active (i.e. when f_trace is set).  At other times we use\n",
    "           PyCode_Addr2Line to calculate the line from the current\n",
    "           bytecode index. */\n",
    "        int f_lineno;               /* Current line number */\n",
    "        int f_iblock;               /* index in f_blockstack */\n",
    "        char f_executing;           /* whether the frame is still executing */\n",
    "        PyTryBlock f_blockstack[CO_MAXBLOCKS]; /* for try and loop blocks */\n",
    "        PyObject *f_localsplus[1];  /* locals+stack, dynamically sized */\n",
    "    } PyFrameObject;\n",
    "    \n",
    " 哇，这么多字段！虽然代码很多，但目前我们需要研究的只有以下这些：\n",
    "\n",
    "|属性|\t描述|\n",
    "|:-:|:-:|\n",
    "|f_back|\t前一个栈帧对象，也就是调用者|\n",
    "|f_builtins|\t内建名字空间|\n",
    "|f_code|\t代码对象|\n",
    "|f_globals|\t全局名字空间|\n",
    "|f_lasti|\t上条已执行字节码指令编号|\n",
    "|f_lineno|\t源码文件行数|\n",
    "|f_locals|\t局部名字空间|\n",
    "|f_localsplus|\t静态存储的局部名字空间和临时栈|\n",
    "\n",
    "看上去，关键字段我们基本都猜对了。至此，我们搞清楚了栈帧对象的结构以及在运行时所起的作用：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_18.png\" width=800 height=600>  \n",
    "\n",
    "其中， f_code 字段保存了当前执行的代码对象，最核心的字节码就在代码对象中。而 f_lasti 字段则保存着上条已执行字节码的编号。虚拟机内部用一个 C 局部变量 next_instr 维护下条字节码的位置，并据此加载下一条待执行的字节码指令，原理跟 CPU 的 **指令指针** 寄存器（ %rip ）一样。\n",
    "\n",
    "另外，注意到 f_back 字段指向前一个栈帧对象，也就是 **调用者** 的栈帧对象。这样一来，栈帧对象按照 **调用关系** 串成一个 **调用链** ！这不是跟 x86 CPU 栈帧布局如出一辙吗？我们先花点时间回顾一下 x86 CPU 栈帧布局与函数调用之间的关系：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_19.png\" width=800 height=600>  \n",
    "\n",
    "x86 体系处理器通过栈维护调用关系，每次函数调用时在栈上分配一个帧用于保存调用上下文以及临时存储。 CPU 中有两个关键寄存器， %rsp 指向当前栈顶，而 %rbp 则指向当前栈帧。每次调用函数时， **调用者** ( Caller )负责准备参数、保存返回地址，并跳转到被调用函数代码；作为 **被调用者** ( Callee )，函数先将当前 %rbp 寄存器压入栈（保存调用者栈帧位置），并将 %rbp 设置为当前栈顶（保存当前新栈帧位置）。由此， %rbp 寄存器与每个栈帧中保存的调用者栈帧地址一起完美地维护了函数调用关系链。\n",
    "\n",
    "现在，我们回过头来继续考察 Python 栈帧对象链以及函数调用之前的关系。请看下面这个例子（ demo.py ）：\n",
    "\n",
    "    pi = 3.14\n",
    "\n",
    "    def square(r):\n",
    "        return r ** 2\n",
    "\n",
    "    def circle_area(r):\n",
    "        return pi * square(r)\n",
    "\n",
    "    def main():\n",
    "        print(circle_area(5))\n",
    "\n",
    "    if __name__ == '__main__':\n",
    "        main()\n",
    "        \n",
    "当 Python 开始执行这个程序时，虚拟机先创建一个栈帧对象，用于执行模块代码对象：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_20.png\" width=800 height=600>  \n",
    "\n",
    "当虚拟机执行到模块代码第 13 行时，发生了函数调用。这时，虚拟机新建一个栈帧对，并开始执行函数 main 的代码对象：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_21.png\" width=800 height=600>  \n",
    "\n",
    "随着函数调用逐层深入，当调用 square 函数时，调用链达到最长：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_22.png\" width=800 height=600>  \n",
    "\n",
    "当函数调用完毕后，虚拟机通过 f_back 字段找到前一个栈帧对象并回到调用者代码中继续执行。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 栈帧获取"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "栈帧对象 PyFrameObject 中保存着 Python 运行时信息，在底层执行流控制以及程序调试中非常有用。那么，在 Python 代码层面，有没有办法获得栈帧对象呢？答案是肯定的。调用标准库 sys 模块中的 _getframe 函数，即可获得当前栈帧对象：\n",
    "\n",
    "    >>> import sys\n",
    "    >>> frame = sys._getframe()\n",
    "    >>> frame\n",
    "    <frame at 0x10e3706a8, file '<stdin>', line 1, code <module>>\n",
    "    >>> dir(frame)\n",
    "    ['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'clear', 'f_back', 'f_builtins', 'f_code', 'f_globals', 'f_lasti', 'f_lineno', 'f_locals', 'f_trace', 'f_trace_lines', 'f_trace_opcodes']\n",
    "    \n",
    "获取栈帧对象后，有什么作用呢？举个例子，我们可以顺着 f_back 字段将调用关系和相关运行时信息打印出来：\n",
    "\n",
    "    import sys\n",
    "\n",
    "    pi = 3.14\n",
    "\n",
    "    def square(r):\n",
    "        frame = sys._getframe()\n",
    "        while frame:\n",
    "            print('#', frame.f_code.co_name)\n",
    "            print('Locals:', list(frame.f_locals.keys()))\n",
    "            print('Globals:', list(frame.f_globals.keys()))\n",
    "            print()\n",
    "\n",
    "            frame = frame.f_back\n",
    "\n",
    "        return r ** 2\n",
    "\n",
    "    def circle_area(r):\n",
    "        return pi * square(r)\n",
    "\n",
    "    def main():\n",
    "        print(circle_area(5))\n",
    "\n",
    "    if __name__ == '__main__':\n",
    "        main()\n",
    "        \n",
    "例子程序在 square 函数中获取栈帧对象，然后逐层输出函数名以及对应的局部名字空间以及全局名字空间。程序执行后，你将看到这样的输出：\n",
    "\n",
    "    # square\n",
    "    Locals: ['r', 'frame']\n",
    "    Globals: ['__name__', '__doc__', '__package__', '__loader__', '__spec__', '__annotations__', '__builtins__', '__file__', '__cached__', 'sys', 'pi', 'square', 'circle_area', 'main']\n",
    "\n",
    "    # circle_area\n",
    "    Locals: ['r']\n",
    "    Globals: ['__name__', '__doc__', '__package__', '__loader__', '__spec__', '__annotations__', '__builtins__', '__file__', '__cached__', 'sys', 'pi', 'square', 'circle_area', 'main']\n",
    "\n",
    "    # main\n",
    "    Locals: []\n",
    "    Globals: ['__name__', '__doc__', '__package__', '__loader__', '__spec__', '__annotations__', '__builtins__', '__file__', '__cached__', 'sys', 'pi', 'square', 'circle_area', 'main']\n",
    "\n",
    "    # <module>\n",
    "    Locals: ['__name__', '__doc__', '__package__', '__loader__', '__spec__', '__annotations__', '__builtins__', '__file__', '__cached__', 'sys', 'pi', 'square', 'circle_area', 'main']\n",
    "    Globals: ['__name__', '__doc__', '__package__', '__loader__', '__spec__', '__annotations__', '__builtins__', '__file__', '__cached__', 'sys', 'pi', 'square', 'circle_area', 'main']\n",
    "\n",
    "    78.5\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### 栈帧获取面试题\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "如果面试官问你，能不能写个函数实现 sys._getframe 一样的功能，你能应付吗？\n",
    "\n",
    "我们知道，Python 程序抛异常时，会将执行上下文带出来，保存在异常中：\n",
    "\n",
    "    >>> try:\n",
    "    ...     1 / 0\n",
    "    ... except Exception as e:\n",
    "    ...     print(e.__traceback__.tb_frame)\n",
    "    ... \n",
    "    <frame at 0x1079713f8, file '<stdin>', line 4, code <module>>\n",
    "    \n",
    "因此，我们自己的 getframe 函数可以这样来写：\n",
    "\n",
    "    def getframe():\n",
    "        try:\n",
    "            1 / 0\n",
    "        except Exception as e:\n",
    "            return e.__traceback__.tb_frame.f_back\n",
    "            \n",
    "请注意， getframe 中通过异常获得的是 getframe 自己的栈帧对象，必须通过 f_back 字段找到调用者的栈帧。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 字节码执行"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Python 虚拟机执行代码对象的代码位于 Python/ceval.c 中，主要函数有两个： PyEval_EvalCodeEx 是通用接口，一般用于函数这样带参数的执行场景； PyEval_EvalCode 是更高层封装，用于模块等无参数的执行场景。\n",
    "\n",
    "    PyObject *\n",
    "    PyEval_EvalCode(PyObject *co, PyObject *globals, PyObject *locals);\n",
    "\n",
    "    PyObject *\n",
    "    PyEval_EvalCodeEx(PyObject *_co, PyObject *globals, PyObject *locals,\n",
    "                      PyObject *const *args, int argcount,\n",
    "                      PyObject *const *kws, int kwcount,\n",
    "                      PyObject *const *defs, int defcount,\n",
    "                      PyObject *kwdefs, PyObject *closure);\n",
    "                      \n",
    "这两个函数最终调用 _PyEval_EvalCodeWithName 函数，初始化栈帧对象并调用 PyEval_EvalFrame 系列函数进行处理。栈帧对象将贯穿代码对象执行的始终，负责维护执行时所需的一切上下文信息。而 PyEval_EvalFrame 系列函数最终调用 _PyEval_EvalFrameDefault 函数，虚拟机执行的秘密就藏在这！\n",
    "\n",
    "    PyObject *\n",
    "    PyEval_EvalFrame(PyFrameObject *f);\n",
    "    PyObject *\n",
    "    PyEval_EvalFrameEx(PyFrameObject *f, int throwflag)\n",
    "\n",
    "    PyObject* _Py_HOT_FUNCTION\n",
    "    _PyEval_EvalFrameDefault(PyFrameObject *f, int throwflag);\n",
    "    \n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_23.png\" width=800 height=600>  \n",
    "\n",
    "虽然 Python/ceval.c 代码量很大(超过 5000 行)，但主体结构并不复杂，很好理解。由于篇幅关系，我们没有办法对 _PyEval_EvalFrameDefault 函数展开深入介绍，只能用伪代码描述该函数的处理结构：\n",
    "\n",
    "    PyObject* _Py_HOT_FUNCTION\n",
    "    _PyEval_EvalFrameDefault(PyFrameObject *f, int throwflag)\n",
    "    {\n",
    "        // 逐条取出字节码来执行\n",
    "        for (;;) {\n",
    "            // 读取下条字节码\n",
    "            // 字节码位于： f->f_code->co_code, 偏移量由 f->f_lasti 决定\n",
    "            opcode, oparg = read_next_byte_code(f);\n",
    "            if (opcode == NULL) {\n",
    "                break;\n",
    "            }\n",
    "\n",
    "            switch (opcode) {\n",
    "                // 加载常量\n",
    "                case LOAD_CONST:\n",
    "                    // ....\n",
    "                    break;\n",
    "                // 加载名字\n",
    "                case LOAD_NAME:\n",
    "                    // ...\n",
    "                    break;\n",
    "                // ...\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "    \n",
    "- 函数主体是一个 for 循环，逐条读入字节码并执行；\n",
    "- for 循环体是一个很大的 switch ，判断字节码指令类型并分别处理；\n",
    "\n",
    "最后，我们以几个典型的字节码例子，演示虚拟机执行的过程。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 顺序执行"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "我们以顺序执行的字节码开始，这类字节码不包含任何跳转指令，逐条执行。下面是一个例子：\n",
    "\n",
    "    pi = 3.14\n",
    "    r = 3\n",
    "    area = pi * r ** 2\n",
    "    \n",
    "对这几行代码进行编译，我们可以得到这样的字节码：\n",
    "\n",
    "      1           0 LOAD_CONST               0 (3.14)\n",
    "                  2 STORE_NAME               0 (pi)\n",
    "\n",
    "      2           4 LOAD_CONST               1 (3)\n",
    "                  6 STORE_NAME               1 (r)\n",
    "\n",
    "      3           8 LOAD_NAME                0 (pi)\n",
    "                 10 LOAD_NAME                1 (r)\n",
    "                 12 LOAD_CONST               2 (2)\n",
    "                 14 BINARY_POWER\n",
    "                 16 BINARY_MULTIPLY\n",
    "                 18 STORE_NAME               2 (area)\n",
    "                 20 LOAD_CONST               3 (None)\n",
    "                 22 RETURN_VALUE\n",
    "                 \n",
    " Python 虚拟机刚开始执行时，准备好栈帧对象用于保存执行上下文，关系如下(全局名字空间等信息省略)：\n",
    " \n",
    "由于 next_instr 初始状态指向字节码开头，虚拟机开始加载第一条字节码指令： LOAD_CONST 0 。字节码分为两部分，分别是 **操作码** ( opcode )和 **操作数** ( oparg ) 。LOAD_CONST 指令表示将常量加载进临时栈，常量下标由操作数给出。LOAD_CONST 指令在 _PyEval_EvalFrameDefault 函数 switch 结构的一个 case 分支中实现：\n",
    " \n",
    " \n",
    "     TARGET(LOAD_CONST) {\n",
    "        PyObject *value = GETITEM(consts, oparg);\n",
    "        Py_INCREF(value);\n",
    "        PUSH(value);\n",
    "        FAST_DISPATCH();\n",
    "    }\n",
    "    \n",
    "虚拟机执行这个指令时，先以操作数为下标从常量表中找到待加载常量并压入位于栈帧对象尾部的临时栈：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_24.png\" width=800 height=600>  \n",
    "\n",
    "接着虚拟机接着执行 STORE_NAME 0 指令，将栈顶元素弹出并保存到局部名字空间，名字下标由操作数给出：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_25.png\" width=800 height=600>  \n",
    "\n",
    "你可能会问，变量赋值为啥不直接存取名字空间，而是到临时栈绕一圈？主要原因在于： Python 字节码只有一个操作数，另一个操作数只能通过临时栈给出。 Python 字节码设计思想跟 CPU **精简指令集** 类似，指令尽量简化，复杂操作由多条指令组合完成。紧接着的两条字节码指令也是类似的，不再赘述。\n",
    "\n",
    "接下的指令就不一条条展开介绍了，我们重点关注每条指令执行完毕后，临时栈的变化：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_26.png\" width=800 height=600>  \n",
    "\n",
    "其中， BINARY_POWER 指令从栈上弹出两个操作数(底数 3 和 指数 2 )进行 **幂运算** ，并将结果 9 压回栈中； BINARY_MULTIPLY 指令则进行 **乘积运算** ，步骤也是类似的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### if 判断"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "接下来，我们继续研究 Python 控制流字节码，先看看最简单的 if 判断语句：\n",
    "\n",
    "    value = 1\n",
    "    if value < 0:\n",
    "        print('negative')\n",
    "    else:\n",
    "        print('positive')\n",
    "        \n",
    "对这几行代码进行编译，我们可以得到这样的字节码：\n",
    "\n",
    "      1           0 LOAD_CONST               0 (1)\n",
    "                  2 STORE_NAME               0 (value)\n",
    "\n",
    "      2           4 LOAD_NAME                0 (value)\n",
    "                  6 LOAD_CONST               1 (0)\n",
    "                  8 COMPARE_OP               0 (<)\n",
    "                 10 POP_JUMP_IF_FALSE       22\n",
    "\n",
    "      3          12 LOAD_NAME                1 (print)\n",
    "                 14 LOAD_CONST               2 ('negative')\n",
    "                 16 CALL_FUNCTION            1\n",
    "                 18 POP_TOP\n",
    "                 20 JUMP_FORWARD             8 (to 30)\n",
    "\n",
    "      4     >>   22 LOAD_NAME                1 (print)\n",
    "                 24 LOAD_CONST               3 ('positive')\n",
    "                 26 CALL_FUNCTION            1\n",
    "                 28 POP_TOP\n",
    "            >>   30 LOAD_CONST               4 (None)\n",
    "                 32 RETURN_VALUE\n",
    "                 \n",
    " 当执行到代码第 2 行 if 语句对应的字节码时，临时栈的变化如下：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_27.png\" width=800 height=600>  \n",
    "\n",
    "POP_JUMP_IF_FALSE 指令将比较结果从栈顶弹出并判断真假，如果为假则跳到 else 分支对应的字节码执行；如果为真则继续执行，最终由 JUMP_FORWARD 指令完成跳转绕过 else 分支(跳过 8 个字节也就是 4 条字节码)。\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_28.png\" width=800 height=600>  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### while 循环"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "最后，我们快速过一遍循环控制结构，以 while 循环为例：\n",
    "\n",
    "    values = [1, 2, 3]\n",
    "\n",
    "    while values:\n",
    "        print(values.pop())\n",
    "        \n",
    "对这几行代码进行编译，我们可以得到这样的字节码：\n",
    "\n",
    "      1           0 LOAD_CONST               0 (1)\n",
    "                  2 LOAD_CONST               1 (2)\n",
    "                  4 LOAD_CONST               2 (3)\n",
    "                  6 BUILD_LIST               3\n",
    "                  8 STORE_NAME               0 (values)\n",
    "\n",
    "      3          10 SETUP_LOOP              20 (to 32)\n",
    "            >>   12 LOAD_NAME                0 (values)\n",
    "                 14 POP_JUMP_IF_FALSE       30\n",
    "\n",
    "      4          16 LOAD_NAME                1 (print)\n",
    "                 18 LOAD_NAME                0 (values)\n",
    "                 20 LOAD_METHOD              2 (pop)\n",
    "                 22 CALL_METHOD              0\n",
    "                 24 CALL_FUNCTION            1\n",
    "                 26 POP_TOP\n",
    "                 28 JUMP_ABSOLUTE           12\n",
    "            >>   30 POP_BLOCK\n",
    "            >>   32 LOAD_CONST               3 (None)\n",
    "                 34 RETURN_VALUE\n",
    "                 \n",
    "经过前面学习，我们轻易便可以读懂这些字节码，并画出以下流程图：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_29.png\" width=800 height=600>  \n",
    "\n",
    "1. SETUP_LOOP 指令拉开循环执行的序幕；\n",
    "2. POP_JUMP_IF_FALSE 指令判断循环条件，成立则往下执行循环体，否则跳过循环体结束循环；\n",
    "3. JUMP_ABSOLUTE 在循环体执行完毕后，跳回循环开头处，再次检查循环条件；\n",
    "4. POP_BLOCK 指令在循环结束后，清理循环环境；"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 总结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "本节我们深入 Python 虚拟机源码，研究虚拟机执行字节码的全过程。虚拟机在执行代码对象前，需要先创建 栈帧对象 ( PyFrameObject )，用于维护运行时的上下文信息。 PyFrameObject 关键信息包括：\n",
    "\n",
    "- 局部名字空间( f_locals )；\n",
    "- 全局名字空间( f_globals )；\n",
    "- 内建名字空间( f_builtins )；\n",
    "- 代码对象( f_code )；\n",
    "- 上条已执行指令编号( f_lasti )；\n",
    "- 调用者栈帧( f_back )；\n",
    "- 静态局部名字空间与临时栈( f_localsplus )；\n",
    "\n",
    "栈帧对象通过 f_back 串成一个 **调用链** ，与 CPU 栈帧调用链有异曲同工之妙。我们还借助 sys 模块成功取得栈帧对象，并在此基础上输出整个函数调用链。\n",
    "\n",
    "    import sys\n",
    "    frame = sys._getframe()\n",
    "\n",
    "最后，我们研究了典型字节码的执行过程，发现这个过程跟 CPU 执行机器指令的过程非常类似。**字节码就像是汇编语言，而虚拟机就像一颗软件 CPU** ！由此可见计算机知识间关系紧密，没有 计算机组成原理 、 操作系统 、 计算机网络 等基础知识加持，程序开发就像在沙子上盖大厦。\n",
    "\n",
    "Python 虚拟机的代码量不小，而本节力求以最通俗的语言将虚拟机的原理讲清楚，因此没有深入源码的细枝末节。鼓励读者保持好奇心，深入到源码中，肯定会有所收获。修炼内功，从源码开始！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## GIL 全局锁，束缚Python的紧箍圈"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "请问 GIL 全局锁的作用是什么？对 Python 程序有什么影响？Python 程序如何充分利用多核 CPU 的计算能力？这几个问题在 Python 面试基本都是必问的，然而有不少候选人对此只是一知半解，知其然而不知其所以然。\n",
    "\n",
    "GIL 作为 Python 虚拟机最大的局限性，对 Python 程序运行性能影响深刻。因此，Python 工程师必须明白虚拟机引入 GIL 的前因后果，避免多线程程序被 Python 虚拟机戴上金箍圈，进而产生不可预料的性能问题。\n",
    "\n",
    "由于 GIL 的存在不可避免，只有想方设法绕过 GIL 的限制才能最大限度地提升 Python 程序的多核执行能力。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### GIL 由来"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "我们先思考一个问题：我们在前面介绍的 list 、 dict 等内建对象是 **线程安全** 的吗？\n",
    "\n",
    "在 Python 层面，list 、dict 等内建对象是线程安全的，这是最基本的常识。研究 list、dict 等内建对象源码时，我们并没有看到任何 **互斥锁** 的痕迹，这多少有点令人意外。以 list 对象 append 方法为例，主要步骤如下：\n",
    "\n",
    "    static int\n",
    "    app1(PyListObject *self, PyObject *v)\n",
    "    {\n",
    "        Py_ssize_t n = PyList_GET_SIZE(self);\n",
    "\n",
    "        assert (v != NULL);\n",
    "        if (n == PY_SSIZE_T_MAX) {\n",
    "            PyErr_SetString(PyExc_OverflowError,\n",
    "                \"cannot add more objects to list\");\n",
    "            return -1;\n",
    "        }\n",
    "\n",
    "        if (list_resize(self, n+1) < 0)\n",
    "            return -1;\n",
    "\n",
    "        Py_INCREF(v);\n",
    "        PyList_SET_ITEM(self, n, v);\n",
    "        return 0;\n",
    "    }\n",
    "    \n",
    "1. 调用 list_resize 将列表长度扩大 1 (第 13-14 行)；\n",
    "2. 将被追加元素设置到末尾位置 (第 16-17 行)；\n",
    "\n",
    "由此可见，append 方法不是一个 **原子操作** 。假设线程 A 调用 append 方法，执行长度扩容后便发生 **线程调度** ；系统唤醒线程 B 开始执行 l[-1] 语句访问 list 末尾元素，会怎样呢？由于 list 长度已经扩容但追加元素尚未设置，线程 B 将得到一个非法对象！\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_30.png\" width=800 height=600>  \n",
    "\n",
    "这种有多线程并发操作导致的 **竞争条件*8 ，一般通过互斥锁加以解决。我们可以为每个 list 对象分配一个互斥锁，当一个 list 操作开始执行前，先获取锁；执行完毕后，再释放锁。进入 **对象锁** 后，竞争条件便消除了。\n",
    "\n",
    "    static int\n",
    "    some_op(PyListObject *self, ...)\n",
    "    {\n",
    "        // acquire lock\n",
    "        lock(self);\n",
    "\n",
    "        // process operation\n",
    "        // ...\n",
    "\n",
    "        // release lock\n",
    "        unlock(self);\n",
    "    }\n",
    "\n",
    "可我们在源码中并没有发现，这又是为什么呢？莫非 Python 采用了其他什么野路子？\n",
    "\n",
    "的确如此。Python 虚拟机维护了一个**全局锁** ，这就是众所周知的 GIL。Python 线程想在虚拟机中执行字节码，必须取得全局锁。这样一来，不管任何时刻，只有一个线程在虚拟机中运行。那么，虚拟机如何交替执行不同线程呢？\n",
    "\n",
    "Python 线程调度实现方式参考了操作系统进程调度中 **时间片** 的思路，只不过将时间片换成 **字节码** 。当一个线程取得 GIL 全局锁并开始执行字节码时，对已执行字节码进行计数。当执行字节码达到一定数量 (比如 100 条) 时，线程主动释放 GIL 全局锁并唤醒其他线程。其他等待 GIL 全局锁的线程取得锁后，将得到虚拟机控制权并开始执行。因此，虚拟机就像一颗软件 CPU ，Python 线程交替在虚拟机上执行：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_31.png\" width=800 height=600>  \n",
    "\n",
    "如图，绿色为当前正在虚拟机中执行的线程；蓝色为等待到虚拟机上执行的线程；黄色为阻塞在 IO 操作上的线程。对于 Running 线程，如果已执行字节码达到一定数量，则自动让出 GIL 并唤醒其他线程，状态变更为 Ready ；如果执行 IO 操作，在执行阻塞型系统调用前先让出 GIL ，状态变更为 IO Blocked 。当 Ready 线程取得 GIL 后，获得虚拟机控制权并开始执行字节码，状态变更为 Running 。 IO Blocked 线程一开始阻塞在系统调用上，当系统调用返回后，状态变更为 Ready ，再次等待 GIL 以便获得虚拟机执行权。\n",
    "\n",
    "那么，Python 为啥采用 GIL 这么简单粗暴的解决方案，而不是对象锁呢？首先，对象锁方案为每个需要保护的对象分配一个互斥锁，内存开销巨大。其次，频繁的加解锁操作严重影响执行效率，特别是 dict 等对象使用频率很高。\n",
    "\n",
    "|线程安全方案|优势|劣势|\n",
    "|:-:|:-:|:-:|\n",
    "|对象锁|\t多线程可并行执行|\t加解锁开销巨大|\n",
    "|GIL|\t无需频繁加解锁|\t多线程只能交替执行|\n",
    "\n",
    "有测试结果表明，引入对象锁获得的多线程 **并行执行** 能力，几乎被加解锁开销完全抵消。而在单线程环境下， GIL 方案优势明显。这样看来，Python 作者们采用 GIL 方案也就理所当然了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### GIL 影响"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "在 GIL 的束缚下，Python 虚拟机同一时刻只能执行一个线程。这是否意味着多线程完全无法优化程序性能呢？由于程序运行特征千差万别，这个问题得分情况讨论。开始之前，我们先来了解两种不同的运行特征：\n",
    "\n",
    "程序在运行时，一般都处于两种状态：\n",
    "\n",
    "- 可执行 状态，包括 Running 以及 Ready 两种情况，这时竞争处理器资源；\n",
    "- 阻塞 状态，一般为等待 IO 处理，这时让出处理器资源；\n",
    "\n",
    "根据程序分别处于 Running 以及 IO Blocked 两种状态的时间占比，可分为两种：\n",
    "\n",
    "- 计算密集型 ，程序执行时大部分时间处于 Running 状态；\n",
    "- IO 密集型 ，程序执行时大部分时间处于 IO Blocked 状态；"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "#### IO 密集型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "典型 IO 密集型 程序时间轴如下，绿色为 Running 状态，黄色为 IO Blocked 状态：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_32.png\" width=800 height=600>  \n",
    "\n",
    "IO 密集型 程序受 GIL 影响相对有限，因为线程在等待 IO 处理时可以让出 GIL 以便其他线程拿到虚拟机执行权：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_33.png\" width=800 height=600>  \n",
    "\n",
    "批量下载网页就是一个典型的 IO 密集型 场景，大部分时间花在等待服务器响应，请求发起以及网页处理所占的时间非常少。因此，一个多线程网络爬虫程序可以极大缩短程序运行时间。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "#### 计算密集型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "诸如科学计算这样的 计算密集型 程序就不一样了，除了开始前读取参数，结束后保存结果，大部分时间都在运算：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_34.png\" width=800 height=600>  \n",
    "\n",
    "计算密集型 程序受 GIL 影响就大了 —— 在 GIL 约束下，虚拟机只能交替执行不同的线程：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_35.png\" width=800 height=600>  \n",
    "\n",
    "由此可见，Python 线程并不具备多核并行执行能力，不能缩短计算密集型程序的运行时间。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 如何提升多核执行能力"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "我们以 **随机数估算$\\pi$** 值 这个典型的 **计算密集型** 场景为例，探索 Python 多核执行能力提升之道：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_36.png\" width=800 height=600>  \n",
    "\n",
    "$\\pi$值可以通过随机试验进行估算，具体做法是：制作一个边长为 1 的正方形，绘出内切扇形，如上图。正方形的面积为 1 ，内切扇形的面积为$\\frac{\\pi r^2 }{4}$ ，即 $\\frac{\\pi}{4}$。随机往正方形发射一个点，该点落在扇形内的概率为$\\frac{\\pi}{4}$ 。现随机往正方形内发射$n$个点，统计其中落在扇形内的点个数$in\\_sectors$，那么$\\pi$值可以这样估算$ \\pi= \\frac{4 * in\\_sectors}{n}$ 。试验次数越大，估算出来的$\\pi$值也就越准确，当然也意味着巨大的计算量。\n",
    "\n",
    "现在，借助 random 模块生成随机数，编写一个试验函数 sample，以试验次数 $n$ 为参数，返回试验次数 $n$ 以及落在扇形内的次数$in\\_sector$：\n",
    "\n",
    "    import random\n",
    "\n",
    "    def sample(n):\n",
    "        in_sectors = 0\n",
    "        # 循环试验n次\n",
    "        for _ in range(n):\n",
    "            # 随机生成一个点，坐标均在0到1之间\n",
    "            x, y = random.random(), random.random()\n",
    "            # 如果该点到原点的距离平方小于1，则在扇形内\n",
    "            if x*x + y*y < 1:\n",
    "                in_sectors += 1\n",
    "        return n, in_sectors\n",
    "        \n",
    "根据试验结果，eval_pi 计算$\\pi$的近似值：\n",
    "\n",
    "    def eval_pi(n, in_sectors):\n",
    "        return 4. * in_sectors / n\n",
    "        \n",
    "现在，我们以单线程模式执行 1 亿次随机试验，estimate_pi 执行耗时 4.59 秒：\n",
    "\n",
    "     def estimate_pi(n):\n",
    "        return eval_pi(*sample(n))   \n",
    "        \n",
    "接下来，我们再以多线程模式执行 1 亿次随机试验，并与单线程模式进行对比。为此我们设计了多线程版的估算函数 estimate_pi_with_threads ，thread_num 参数为线程数：\n",
    "\n",
    "    from queue import Queue\n",
    "    from threading import Thread\n",
    "\n",
    "    def estimate_pi_with_threads(n, thread_num):\n",
    "        # 用于收集线程执行结果的队列\n",
    "        queue = Queue()\n",
    "\n",
    "        # 线程执行函数\n",
    "        def thread_routine():\n",
    "            # 将试验次数n分为thread_num份\n",
    "            # 执行试验并将试验结果放入队列\n",
    "            queue.put(sample(n // thread_num))\n",
    "\n",
    "        # 创建执行线程\n",
    "        threads = [\n",
    "            Thread(target=thread_routine)\n",
    "            for _ in range(thread_num)\n",
    "        ]\n",
    "\n",
    "        # 启动执行线程\n",
    "        for thread in threads:\n",
    "            thread.start()\n",
    "\n",
    "        # 从任务队列收集试验结果\n",
    "        total_n, total_in_sectors = 0, 0\n",
    "        for thread in threads:\n",
    "            n, in_sectors = queue.get()\n",
    "            total_n += n\n",
    "            total_in_sectors += in_sectors\n",
    "\n",
    "        # 回收线程资源\n",
    "        for thread in threads:\n",
    "            thread.join()\n",
    "\n",
    "        return eval_pi(total_n, total_in_sectors)\n",
    "        \n",
    "estimate_pi_with_threads 估算函数执行 1 亿次试验耗时 4.63 秒，多线程对程序没有任何提升。\n",
    "\n",
    "那么，是否意味着 Python 程序无法充分利用多核 CPU 的执行能力呢？\n",
    "\n",
    "肯定不是。虽然我们没有办法避免 GIL 的影响，但是我们可以想法设法绕过它，例如采用 **多进程模式** 。在 Python 程序中，每个进程独立运行一个虚拟机。因此，不同 Python 进程可以在多核 CPU 上并行运行，不受 GIL 限制。\n",
    "\n",
    "最后，我们再以 **多进程模式** 执行 1 亿次随机试验，看执行效率是否如预期有所提升。为此我们设计了多进程版的估算函数 estimate_pi_with_processes ，process_num 参数为子进程数：\n",
    "\n",
    "    from multiprocessing import Process, Queue as ProcessQueue\n",
    "\n",
    "    def estimate_pi_with_processes(n, process_num):\n",
    "        # 用于收集子进程执行结果的队列\n",
    "        queue = ProcessQueue()\n",
    "\n",
    "        # 子进程执行函数\n",
    "        def process_routine():\n",
    "            # 将试验次数n分为process_num份\n",
    "            # 执行试验并将试验结果放入队列\n",
    "            queue.put(sample(n // process_num))\n",
    "\n",
    "        # 创建执行进程\n",
    "        processes = [\n",
    "            Process(target=process_routine)\n",
    "            for _ in range(process_num)\n",
    "        ]\n",
    "\n",
    "        # 启动执行进程\n",
    "        for process in processes:\n",
    "            process.start()\n",
    "\n",
    "        # 从任务队列收集试验结果\n",
    "        total_n, total_in_sectors = 0, 0\n",
    "        for process in processes:\n",
    "            n, in_sectors = queue.get()\n",
    "            total_n += n\n",
    "            total_in_sectors += in_sectors\n",
    "\n",
    "        # 回收子进程资源\n",
    "        for process in processes:\n",
    "            process.join()\n",
    "\n",
    "        return eval_pi(total_n, total_in_sectors)\n",
    "    \n",
    "estimate_pi_with_processes 估算函数执行 1 亿次试验耗时 2.56 秒，几乎比单线程版快了一倍！如果增加子进程数，程序执行速度还可以进一步提升。需要特别注意，超出 CPU 核数的进程数则没有任何意义了。\n",
    "\n",
    "|模式|试验函数|耗时 (秒)|\n",
    "|:-:|:-:|:-:|\n",
    "|单线程|\testimate_pi(100000000)\t|4.59|\n",
    "|多线程|\testimate_pi_with_threads(100000000, 2)\t|4.63|\n",
    "|多进程|\testimate_pi_with_processes(100000000, 2)\t|2.56|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## 模块动态加载， import 背后哪些事儿"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "我们知道 import 关键字用于导入模块，例如：\n",
    "\n",
    "    import demo\n",
    "\n",
    "此外，import 语句还有其他几种变体，例如：\n",
    "\n",
    "    import demo as d\n",
    "\n",
    "    from demo import value\n",
    "    from demo import value as v\n",
    "    \n",
    "那么，Python 模块加载的过程是怎样的呢？不同 import 语句都有哪些异同，背后具体执行了什么动作？Python 又是如何找到被导入模块的呢？带着这些问题，我们开始学习 Python 的 **模块加载** 机制。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 字节码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "透过字节码，我们可以洞悉 Python 执行语句的全部秘密。因此，我们从研究 import 语句字节码入手，逐步深入研究模块的加载过程。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "#### import"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "以最基本的 import 语句为例：\n",
    "\n",
    "    import demo\n",
    "    \n",
    "借助 dis 模块，我们将这个语句反编译，得到以下字节码：\n",
    "\n",
    "      1           0 LOAD_CONST               0 (0)\n",
    "                  2 LOAD_CONST               1 (None)\n",
    "                  4 IMPORT_NAME              0 (demo)\n",
    "                  6 STORE_NAME               0 (demo)\n",
    "                  8 LOAD_CONST               1 (None)\n",
    "                 10 RETURN_VALUE\n",
    "\n",
    "我们重点关注前 4 条字节码，看它们在 Python 虚拟机中是如何执行的：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_37.png\" width=800 height=600>  \n",
    "\n",
    "1. 前 2 条字节码执行完毕后，0 以及 None 这两个常量被加载到栈中；\n",
    "2. 顾名思义，IMPORT_NAME 指令负责加载模块，模块名由操作数指定，其他参数从栈上取；模块加载完毕后，模块对象便保存在栈顶；\n",
    "3. 最后，STORE_NAME 指令从栈顶取出模块对象并保存到局部名字空间中；\n",
    "\n",
    "至此，Python 模块动态加载的秘密已经浮出水面了。在字节码层面，IMPORT_NAME 负责加载模块， **模块名** 由操作数指定，其他参数来源于 **运行栈** 。虽然 IMPORT_NAME 指令还没来得及研究，成就感也是满满的呢！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "#### import as"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "开始研究 IMPORT_NAME 指令实现细节前，一鼓作气将其他几种 import 语句变体拿下。先研究 impot as 语句：\n",
    "\n",
    "    import demo as d\n",
    "    \n",
    "同样用 dis 对语句进行反编译，我们得到以下字节码：\n",
    "\n",
    "      1           0 LOAD_CONST               0 (0)\n",
    "                  2 LOAD_CONST               1 (None)\n",
    "                  4 IMPORT_NAME              0 (demo)\n",
    "                  6 STORE_NAME               1 (d)\n",
    "                  8 LOAD_CONST               1 (None)\n",
    "                 10 RETURN_VALUE\n",
    "         \n",
    "这段字节码跟前一段几乎一模一样，区别只是 STORE_NAME 指令，它用换个名字来保存被加载模块：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_38.png\" width=800 height=600>  \n",
    "\n",
    "因此，这个 import 语句变体其实等价于：\n",
    "\n",
    "    import demo\n",
    "    d = demo\n",
    "    del demo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "#### from import"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "现在再接再厉，拿下 from import 语句：\n",
    "\n",
    "    from demo import value\n",
    "    \n",
    "同样用 dis 对语句进行反编译，我们得到以下字节码：\n",
    "\n",
    "      1           0 LOAD_CONST               0 (0)\n",
    "                  2 LOAD_CONST               1 (('value',))\n",
    "                  4 IMPORT_NAME              0 (demo)\n",
    "                  6 IMPORT_FROM              1 (value)\n",
    "                  8 STORE_NAME               1 (value)\n",
    "                 10 POP_TOP\n",
    "                 12 LOAD_CONST               2 (None)\n",
    "                 14 RETURN_VALUE\n",
    "                 \n",
    " 我们看到一个新面孔—— IMPORT_FROM 指令。该指令从栈顶模块中取出指定名字，并保存于栈顶。\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_39.png\" width=800 height=600>  \n",
    "\n",
    "注意到，value 以 **元组** 的形式保存于栈顶，IMPORT_NAME 指令如果发现 value 为 demo 模块的子模块，将同时加载 value 子模块。此外，IMPORT_FROM 与 STORE_NAME 这两个指令相互配合，从模块中取出给定名字并保存。如果 from import 语句一次性导入多个名字，字节码将包含多个由 IMPORT_FROM 和 STORE_NAME 组成的指令对：\n",
    "\n",
    "    from demo import value, data\n",
    "    \n",
    "如下：\n",
    "\n",
    "      1           0 LOAD_CONST               0 (0)\n",
    "                  2 LOAD_CONST               1 (('value', 'data'))\n",
    "                  4 IMPORT_NAME              0 (demo)\n",
    "                  6 IMPORT_FROM              1 (value)\n",
    "                  8 STORE_NAME               1 (value)\n",
    "                 10 IMPORT_FROM              2 (data)\n",
    "                 12 STORE_NAME               2 (data)\n",
    "                 14 POP_TOP\n",
    "                 16 LOAD_CONST               2 (None)\n",
    "                 18 RETURN_VALUE\n",
    "                 \n",
    " 这也是 IMPORT_FROM 指令不将模块对象从栈顶弹出的原因——只有所有需要导入的名字均导入完毕，模块对象才可从栈顶弹出，这个使命正是由紧接着的 POP_TOP 指令完成！\n",
    "\n",
    "综上所述，from import 语句实际上等价于：\n",
    "\n",
    "    import demo\n",
    "    value = demo.value\n",
    "    del demo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "#### from import as"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "from import as 语句与 from import 语句的关系，跟 import as 语句与 import 语句的关系一样。这两种语句的差别只在 STORE_NAME 字节码的操作数上，而操作数直接决定了被导入对象以什么名字在局部名字空间中保存。from import as 语句示例以及对应的字节码分别如下，相信轻易即可看懂，就不再赘述了：\n",
    "\n",
    "from demo import value as v\n",
    "\n",
    "如下：\n",
    "\n",
    "      1           0 LOAD_CONST               0 (0)\n",
    "                  2 LOAD_CONST               1 (('value',))\n",
    "                  4 IMPORT_NAME              0 (demo)\n",
    "                  6 IMPORT_FROM              1 (value)\n",
    "                  8 STORE_NAME               2 (v)\n",
    "                 10 POP_TOP\n",
    "                 12 LOAD_CONST               2 (None)\n",
    "                 14 RETURN_VALUE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 模块加载流程"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "现在回过头来看 Python 虚拟机是如何执行 IMPORT_NAME 指令的，从中便可洞悉 Python **模块动态加载机制**。\n",
    "\n",
    "IMPORT_NAME 字节码指令在 Include/opcode.h 头文件中定义，其后紧挨着 IMPORT_FROM 指令：\n",
    "\n",
    "#define IMPORT_NAME             108\n",
    "#define IMPORT_FROM             109\n",
    "\n",
    "Python 虚拟机实现位于 Python/ceval.c 源文件中，IMPORT_NAME 指令的处理逻辑也在其中(第 2595 行)：\n",
    "\n",
    "        TARGET(IMPORT_NAME) {\n",
    "            PyObject *name = GETITEM(names, oparg);\n",
    "            PyObject *fromlist = POP();\n",
    "            PyObject *level = TOP();\n",
    "            PyObject *res;\n",
    "            res = import_name(f, name, fromlist, level);\n",
    "            Py_DECREF(level);\n",
    "            Py_DECREF(fromlist);\n",
    "            SET_TOP(res);\n",
    "            if (res == NULL)\n",
    "                goto error;\n",
    "            DISPATCH();\n",
    "        }\n",
    "        \n",
    "1. 第 2 行，根据字节码 **操作数** 取出待加载 模块名 ；\n",
    "2. 第 3 行，从栈顶弹出 fromlist 参数，参数是一个元组，列举了需要加载的潜在 **子模块** ；\n",
    "3. 第 4 行，从栈顶弹出 level 参数；\n",
    "4. 第 6 行，调用 import_name 函数完成模块加载工作；\n",
    "5. 第 7-9 行，释放参数并将加载到的 **模块对象** 保存到栈顶；\n",
    "\n",
    "import_name 函数则调用位于 Python/import.c 的 PyImport_ImportModuleLevelObject 函数，接口如下：\n",
    "\n",
    "    PyObject *\n",
    "    PyImport_ImportModuleLevelObject(PyObject *name, PyObject *globals,\n",
    "                                     PyObject *locals, PyObject *fromlist,\n",
    "                                     int level);\n",
    "                                 \n",
    " 函数大部分参数我们已经很熟悉了，但 globals 和 locals 这两个名字空间有什么作用呢？Python 导入模块时可以使用相对路径，这时需要根据 import 语句所在模块计算绝对路径，而模块名则保存在 globals 名字空间中：\n",
    " \n",
    "     from .a.b import c\n",
    "     \n",
    "     \n",
    " 我不打算事无巨细地介绍 Python/import.c 中的源码，毕竟超过 2000 行的代码量需要相当的篇幅才能讲解清楚。接下来，我力求以最简洁的 Python 语言，描述清楚模块的加载流程。鼓励学有余力的童鞋，到源码中探究一番。\n",
    "\n",
    "解析得到绝对模块名后，Python 便开始查找并加载模块。那么，如果模块已经加载过了，Python 如何处理呢呢？说来也简单，Python 内部用一个 dict 对象记录所有已经加载的模块，这个 dict 位于 sys 模块中：\n",
    "\n",
    "    >>> import sys\n",
    "    >>> for name, module in sys.modules.items():\n",
    "    ...     print(name)\n",
    "    ... \n",
    "    sys\n",
    "    builtins\n",
    "    # ...\n",
    "    \n",
    "想要加载被导入模块，Python 需要找到模块代码的具体位置，这便是 Python **模块搜索** 过程。Python 在内部维护了一个模块搜索路径的列表，同样位于 sys 模块内：\n",
    "\n",
    "    >>> import sys\n",
    "    >>> for path in sys.path:\n",
    "    ...     print(repr(path))\n",
    "    ... \n",
    "    ''\n",
    "    '/Users/fasion/opt/pythons/python3/lib/python37.zip'\n",
    "    '/Users/fasion/opt/pythons/python3/lib/python3.7'\n",
    "    '/Users/fasion/opt/pythons/python3/lib/python3.7/lib-dynload'\n",
    "    '/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7'\n",
    "    '/Users/fasion/opt/pythons/python3/lib/python3.7/site-packages'\n",
    "    \n",
    "Python 遍历每个路径，直到发现目标模块。如果遍历完所有路径还是没找到目标模块，Python 便只好抛异常了。目标模块代码找到后，Python 对代码进行编译，生成 PyCodeObject 。如果存在 pyc 文件，则可以省略编译过程，直接从 pyc 文件中加载 PyCodeObject 。以导入 demo 模块为例：\n",
    "\n",
    "    # 读取模块代码\n",
    "    text = read('demo.py')\n",
    "\n",
    "    # 编译模块代码\n",
    "    code = compile(text, 'demo.py', 'exec')\n",
    "    \n",
    "至此，Python 得到了代表模块逻辑的 **代码** 对象 PyCodeObject 。接着，Python 创建一个全新的 **模块** 对象。模块对象在 Python 内部由 PyModuleObject 结构体表示，位于 Objects/moduleobject.c 。 **模块** 对象也是内建对象中的一种，有兴趣的童鞋参照 **内建对象** 部分的思路研究源码，在此不再展开。\n",
    "\n",
    "由于需要直接创建 **模块对象** 的场景极少，Python 没有将 **模块类型** 暴露出来，这跟其他内置对象类型略有差别：\n",
    "\n",
    "    >>> int\n",
    "    <class 'int'>\n",
    "    >>> module\n",
    "    Traceback (most recent call last):\n",
    "      File \"<stdin>\", line 1, in <module>\n",
    "    NameError: name 'module' is not defined\n",
    "    \n",
    "根据 对象模型 部分学到的知识，还有另一条路可以找到 模块类型 对象——通过已存在的 模块 对象顺藤摸瓜！由于每个 Python 程序都包含一个 __main__ 模块，这就是最合适的媒介了：\n",
    "\n",
    "    >>> import __main__\n",
    "    >>> __main__\n",
    "    <module '__main__' (built-in)>\n",
    "    \n",
    "**模块类型** 对象便位于 **模块** 对象的 ob_type 字段，通过 __class__ 属性即可获取：\n",
    "\n",
    "    >>> module = __main__.__class__\n",
    "    >>> module\n",
    "    <class 'module'>\n",
    "    \n",
    "Python 内部便是通过 **模块类型** 对象创建实例对象的，只需提供 **模块名** 以及 **模块文档信息** ：\n",
    "\n",
    "    >>> demo = module('demo', 'A test module')\n",
    "    >>> help(demo)\n",
    "    >>> dir(demo)\n",
    "    ['__doc__', '__loader__', '__name__', '__package__', '__spec__']\n",
    "    >>> demo.__doc__\n",
    "    'A test module'\n",
    "    \n",
    "至此，我们得到一个全新的 模块 对象，尽管这个模块对象还是个空架子。根据前面章节的内容，我们知道 模块 的 属性空间 由一个 dict 对象实现，这个 dict 可通过 __dict__ 属性找到：\n",
    "\n",
    "    >>> demo.__dict__\n",
    "    {'__name__': 'demo', '__doc__': 'A test module', '__package__': None, '__loader__': None, '__spec__': None}\n",
    "    >>> demo.a\n",
    "    Traceback (most recent call last):\n",
    "      File \"<stdin>\", line 1, in <module>\n",
    "    AttributeError: module 'demo' has no attribute 'a'\n",
    "    >>> demo.__dict__['a'] = 1\n",
    "    >>> demo.a\n",
    "    1\n",
    "    \n",
    "接着，Python 执行模块代码对象，完成模块初始化：\n",
    "\n",
    "    >>> exec(code, demo.__dict__, demo.__dict__)\n",
    "    \n",
    "注意到，模块 属性空间 作为 全局名字空间 以及 局部名字空间 传给了 exec 函数。\n",
    "\n",
    "假设 demo.py 只包含一行代码，请大家自行脑补字节码在 Python 虚拟机执行的步骤以及结果：\n",
    "\n",
    "    value = 1\n",
    "    \n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_40.png\" width=800 height=600> \n",
    "\n",
    "最后，别忘了将 demo 模块 对象保存到 sys.modules ，以避免不必要的重复加载：\n",
    "\n",
    "    >>> import sys\n",
    "    >>> sys.modules['demo'] = demo\n",
    "    \n",
    "这便是 Python **模块动态加载** 的全过程，其实也并不复杂，对吧？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 模块搜索方式"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "曾几何时，我们对 ModuleNotFoundError 异常谈虎色变。明明已经准备好了模块代码，Python 为何不认呢？全面掌握 **模块加载** 机制，特别是 **模块搜索** 方式后，我们内心便不再畏惧。\n",
    "\n",
    "Python 模块搜索路径保存于 sys.path 列表中，遇到 ModuleNotFoundError 异常首先要排查 sys.path 是否包含目标模块所在目录路径。如果目标模块所在路径不在 sys.path 中，则需要将其加入：\n",
    "\n",
    "    # 路径追加到末尾，Python最后才搜索该路径\n",
    "    sys.path.append('/some/path')\n",
    "\n",
    "    # 路径插入到头部，Python则最先搜索该路径\n",
    "    sys.path.insert(0, '/some/path')\n",
    "    \n",
    "需要特别注意，同一路径不能重复多次加入，不然将影响 Python 的搜索效率，甚至导致 内存泄露 。我曾经帮一个初学者排查内存泄露问题，最后定位到这样的代码：\n",
    "\n",
    "    def some_function():\n",
    "        sys.path.insert(0, '/some/path')\n",
    "        import xxxx\n",
    "\n",
    "        # ...\n",
    "        \n",
    "这意味着函数每调用一次，sys.path 便增加一个元素！当函数被频繁调用，sys.path 列表最终将耗尽程序内存！\n",
    "\n",
    "此外，在程序代码中写死模块搜索路径的方式并不可取。更优雅的方式是通过 PYTHONPATH 环境变量指定：\n",
    "\n",
    "    PYTHONPATH=/some/path python xxxx.py\n",
    "\n",
    "这样一来，程序启动后给定路径 /some/path 便在 sys.path 列表中了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## 用字节码彻底征服面试官"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**请问 Python 程序是怎么运行的？是编译成机器码后在执行的吗？试着与 C、C++ 、Java、Shell 等常见语言比较说明。**\n",
    "\n",
    "不少初学者对 Python 存在误解，以为它是类似 Shell 的解释性脚本语言，其实并不是。虽然执行 Python 程序的 python 命令也被称为 Python 解释器，但它其实包含一个 编译器 和一个 虚拟机 。\n",
    "\n",
    "当我们在命令行敲下 python xxxx.py 时，python 命令中的编译器首先登场，将 Python 代码编译成 代码 对象。代码 对象包含 字节码 以及执行字节码所需的 名字 以及 常量 。\n",
    "\n",
    "当编译器完成编译动作后，接力棒便传给 虚拟机 。虚拟机 维护执行上下文，逐行执行 字节码 指令。执行上下文中最核心的 名字空间 ，便是由 虚拟机 维护的。\n",
    "\n",
    "因此，Python 程序的执行原理其实更像 Java，可以用两个词来概括—— 虚拟机和字节码 。不同的是，Java 编译器 javac 与 虚拟机 java 是分离的，而 Python 将两者整合成一个 python 命令。此外，Java 程序执行前必须先完整编译，而 Python 则允许程序启动后再编译并加载需要执行的模块。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**pyc 文件保存什么东西，有什么作用？**\n",
    "\n",
    "Python 程序执行时需要先由 编译器 编译成 代码 对象，然后再交由 虚拟机 来执行。不管程序执行多少次，只要源码没有变化，编译后得到的代码对象就肯定是一样的。因此，Python 将代码对象序列化并保存到 pyc 文件中。当程序再次执行时，Python 直接从 pyc 文件中加载代码对象，省去编译环节。当然了，当 py 源码文件改动后，pyc 文件便失效了，这时 Python 必须重新编译 py 文件。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**如何查看 Python 程序的字节码？**\n",
    "\n",
    "Python 标准库中的 dis 模块，可以对**代码**对象以及 **函数** 对象进行反编译，并显示其中的 **字节码**。\n",
    "\n",
    "例如，对于函数 add ，通过 __code__ 字段取到它的 **代码** 对象，并调用 dis 进行反编译：\n",
    "\n",
    "    >>> def add(x, y):\n",
    "    ...     return x + y\n",
    "    ...\n",
    "    >>> dis.dis(add.__code__)\n",
    "      2           0 LOAD_FAST                0 (x)\n",
    "                  2 LOAD_FAST                1 (y)\n",
    "                  4 BINARY_ADD\n",
    "                  6 RETURN_VALUE\n",
    "                  \n",
    "当然了，直接将 函数 对象传给 dis 也可以：\n",
    "\n",
    "    >>> dis.dis(add)\n",
    "      2           0 LOAD_FAST                0 (x)\n",
    "                  2 LOAD_FAST                1 (y)\n",
    "                  4 BINARY_ADD\n",
    "                  6 RETURN_VALUE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**Python 中变量交换有两种不同的写法，示例如下。这两种写法有什么区别吗？那种写法更好？**\n",
    "\n",
    "    # 写法一\n",
    "    a, b = b, a\n",
    "\n",
    "    # 写法二\n",
    "    tmp = a\n",
    "    a = b\n",
    "    b = tmp\n",
    "    \n",
    "这两种写法都能实现变量交换，表面上看第一种写法更加简洁明了，似乎更优。那么，在优雅的外表下是否隐藏着不为人知的性能缺陷呢？想要找打答案，唯一的途径是研究字节码：\n",
    "\n",
    " - 写法一\n",
    " \n",
    "      1           0 LOAD_NAME                0 (b)\n",
    "                  2 LOAD_NAME                1 (a)\n",
    "                  4 ROT_TWO\n",
    "                  6 STORE_NAME               1 (a)\n",
    "                  8 STORE_NAME               0 (b)\n",
    "- 写法二\n",
    "\n",
    "      1           0 LOAD_NAME                0 (a)\n",
    "                  2 STORE_NAME               1 (tmp)\n",
    "\n",
    "      2           4 LOAD_NAME                2 (b)\n",
    "                  6 STORE_NAME               0 (a)\n",
    "\n",
    "      3           8 LOAD_NAME                1 (tmp)\n",
    "                 10 STORE_NAME               2 (b)\n",
    "                 \n",
    " 从字节码上看，第一种写法需要的指令条目也更少：先将两个变量依次加载到栈，然后一条 ROT_TWO 指令将栈中的两个变量交换，最后再将变量依次写回去。注意到，变量加载的顺序与 = 右边一致，写回顺序与 = 左边一致。\n",
    " \n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_41.png\" width=800 height=600> \n",
    "\n",
    "而且，ROT_TWO 指令只是将栈顶两个元素交换位置，执行起来比 LOAD_NAME 和 STORE_NAME 都要快。\n",
    "\n",
    "至此，我们可以得到结论了—— 第一种变量交换写法更优 ：\n",
    "\n",
    "- 代码简洁明了，不拖泥带水；\n",
    "- 不需要辅助变量 tmp ，节约内存；\n",
    "- ROT_TWO 指令比一个 LOAD_NAME STORE_NAME 指令对更有优势，执行效率更高；"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**请解释 is 和 == 这两个操作的区别。**\n",
    "    a is b\n",
    "    a == b\n",
    "    \n",
    "我们知道 is 是 **对象标识符** ( object identity )，判断两个引用 **是不是同一个对象** ，等价于 id(a) == id(b) ；而 == 操作符判断两个引用 **是不是相等** ，等价于调用魔法方法 a.__eq__(b) 。因此，== 操作符可以通过 __eq__ 魔法方法进行覆写( overriding )，而 is 操作符无法覆写。\n",
    "\n",
    "从字节码上看，这两个语句也很接近，区别仅在比较指令 COMPARE_OP 的 **操作数** 上：\n",
    "\n",
    "-  a is b\n",
    "\n",
    "      1           0 LOAD_NAME                0 (a)\n",
    "                  2 LOAD_NAME                1 (b)\n",
    "                  4 COMPARE_OP               8 (is)\n",
    "                  6 POP_TOP\n",
    "- a == b\n",
    "\n",
    "     1           0 LOAD_NAME                0 (a)\n",
    "              2 LOAD_NAME                1 (b)\n",
    "              4 COMPARE_OP               2 (==)\n",
    "              6 POP_TOP\n",
    "              \n",
    "COMPARE_OP 指令处理逻辑在 Python/ceval.c 源文件中实现，关键函数是 cmp_outcome ：\n",
    "\n",
    "     static PyObject *\n",
    "    cmp_outcome(int op, PyObject *v, PyObject *w)\n",
    "    {\n",
    "        int res = 0;\n",
    "        switch (op) {\n",
    "        case PyCmp_IS:\n",
    "            res = (v == w);\n",
    "            break;\n",
    "        // ...\n",
    "        default:\n",
    "            return PyObject_RichCompare(v, w, op);\n",
    "        }\n",
    "        v = res ? Py_True : Py_False;\n",
    "        Py_INCREF(v);\n",
    "        return v;\n",
    "    }\n",
    "    \n",
    " 源码表明，is 操作操作符仅需比较两个对象的 **地址** (指针)是不是相同(第 7 行)。包括 == 在内的其他比较操作，则需要调用 PyObject_RichCompare 函数，而 PyObject_RichCompare 将进一步调用对象的魔法方法进行判断。   \n",
    "这个例子，a 和 b 均引用同一个对象，is 和 == 操作均返回 True ：\n",
    "\n",
    "    >>> a = 1024\n",
    "    >>> b = a\n",
    "    >>> a is b\n",
    "    True\n",
    "    >>> a == b\n",
    "    True\n",
    "    \n",
    "\n",
    "另一个例子，a 和 b 引用不同的两个 整数 对象，整数对象数值均为 1024 ：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_42.png\" width=800 height=600> \n",
    "\n",
    "显然，由于背后对象是不同的， is 操作结果是 False ；而对象值相同，== 操作结果是 True 。\n",
    "\n",
    "一般而言，对象与自身比较，结果是相等的。因此，如果 a is b 结果为 True ，那么 a == b 多半也是 True 。但这并不是一个永远成立的铁律，至少存在一个例外—— **浮点数** nan 。nan 是一个特殊的 浮点数 ，用于表示 异常值 ，即不存在或者非法的值。不管 nan 跟任何浮点(包括自身)做何种数学比较，结果均为 False 。\n",
    "\n",
    "    >>> a = float('nan')\n",
    "    >>> b = a\n",
    "    >>> a is b\n",
    "    True\n",
    "    >>> a == b\n",
    "    False\n",
    "    \n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_43.png\" width=800 height=600> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**在 Python 中与 None 比较时，为什么要用 is None 而不是 == None ？**\n",
    "\n",
    "None 是一种特殊的内建对象，它是 **单例** 对象，在整个程序中只有一个实例。因此，如果一个变量是 None ，它和 None 一定指向同一个实例，内存地址肯定也相同。这就是要用 is 操作符与 None 比较的原因。\n",
    "\n",
    "Python 中的 == 操作符对两个对象进行相等性比较，背后调用 __eq__ 魔法方法。在自定义类中，__eq__ 方法可以被覆写，实现：\n",
    "\n",
    "    >>> class Foo(object):\n",
    "    ...     def __eq__(self, other):\n",
    "    ...         return True\n",
    "    ...\n",
    "    >>> f = Foo()\n",
    "    >>> f is None\n",
    "    False\n",
    "    >>> f == None\n",
    "    True\n",
    "    \n",
    "由于 is None 只需比较对象的 **内存地址** ，而 == None 需要调用对象的魔法函数进行判断，因此 is None 更有性能优势。下面这个程序尖锐地揭开这两种比较间微妙的性能差距：\n",
    "\n",
    "    import time\n",
    "\n",
    "    class Foo(object):\n",
    "\n",
    "        def __eq__(self, other):\n",
    "            return False\n",
    "\n",
    "    def compare_with_eq(n):\n",
    "        nones = 0\n",
    "\n",
    "        f = Foo()\n",
    "        for _ in range(n):\n",
    "            if f == None:\n",
    "                nones += 1\n",
    "\n",
    "        return nones\n",
    "\n",
    "    def compare_with_is(n):\n",
    "        nones = 0\n",
    "\n",
    "        f = Foo()\n",
    "        for _ in range(n):\n",
    "            if f is None:\n",
    "                nones += 1\n",
    "\n",
    "        return nones\n",
    "\n",
    "    def test(func, n):\n",
    "        start_ts = time.time()\n",
    "\n",
    "        func(n)\n",
    "\n",
    "        print('call function {name} with {n} times in {seconds:.2f}s'.format(\n",
    "            name=func.__name__,\n",
    "            n=n,\n",
    "            seconds=time.time()-start_ts,\n",
    "        ))\n",
    "\n",
    "    if __name__ == '__main__':\n",
    "        N = 100000000\n",
    "\n",
    "        test(compare_with_eq, N)\n",
    "        test(compare_with_is, N)\n",
    "        \n",
    "例子程序将自定义类 Foo 对象与 None 进行一亿次比较，在我的 MacBook 上表现分别如下：\n",
    "- == None 方式耗时大概 16 秒；\n",
    "- is None 方式耗时大概 4 秒；\n",
    "\n",
    "两者性能相差整整 4倍！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T05:00:04.223767Z",
     "start_time": "2021-03-29T04:59:48.152971Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "class Foo(object):\n",
    "\n",
    "    def __eq__(self, other):\n",
    "        return False\n",
    "\n",
    "def compare_with_eq(n):\n",
    "    nones = 0\n",
    "\n",
    "    f = Foo()\n",
    "    for _ in range(n):\n",
    "        if f == None:\n",
    "            nones += 1\n",
    "\n",
    "    return nones\n",
    "\n",
    "def compare_with_is(n):\n",
    "    nones = 0\n",
    "\n",
    "    f = Foo()\n",
    "    for _ in range(n):\n",
    "        if f is None:\n",
    "            nones += 1\n",
    "\n",
    "    return nones\n",
    "\n",
    "def test(func, n):\n",
    "    start_ts = time.time()\n",
    "\n",
    "    func(n)\n",
    "\n",
    "    print('call function {name} with {n} times in {seconds:.2f}s'.format(\n",
    "        name=func.__name__,\n",
    "        n=n,\n",
    "        seconds=time.time()-start_ts,\n",
    "    ))\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    N = 100000000\n",
    "\n",
    "    test(compare_with_eq, N)\n",
    "    test(compare_with_is, N)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**请问以下程序输出什么？为什么？**\n",
    "\n",
    "    # data.py\n",
    "\n",
    "    values = [1, 2, 3]\n",
    "    import data\n",
    "    from data import values\n",
    "\n",
    "    values.pop()\n",
    "    print(data.values)\n",
    "\n",
    "    values = [3.14, 2.71]\n",
    "    print(data.values)\n",
    "    \n",
    "在 **模块加载机制** 一节，我们学习到 from import 语句的原理：先加载模块对象，在从模块对象中取出指定属性，并引入到当前 **局部名字空间** 。与 from data import values 相对应的字节码如下：\n",
    "\n",
    "      1           0 LOAD_CONST               0 (0)\n",
    "                  2 LOAD_CONST               1 (('values',))\n",
    "                  4 IMPORT_NAME              0 (data)\n",
    "                  6 IMPORT_FROM              1 (values)\n",
    "                  8 STORE_NAME               1 (values)\n",
    "                 10 POP_TOP\n",
    "                 \n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_44.png\" width=800 height=600> \n",
    "\n",
    "局部名字空间中 values 指向的 list 对象与模块 data 中 values 是同一个。对局部名字空间的 values 进行操作，等价于对 data 模块中的 values 进行操作。因此，第一个 print 语句输出 [1, 2] 。\n",
    "\n",
    "当代码第 7 行对局部名字空间中的 values 进行重新赋值，指向一个新的 list 对象后，虚拟机中的状态是这样的：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_03\\C3_45.png\" width=800 height=600> \n",
    "\n",
    "这时，局部名字空间中的 values 与模块中的保存独立，不再有任何关联了。因此，整个程序输出内容如下：\n",
    "\n",
    "    [1, 2]\n",
    "    [1, 2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## 函数对象诞生记"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "函数 ，作为计算机程序中 抽象执行流程 的基本单位，在 功能分解 、 代码复用 等方面发挥着至关重要的作用。Python 中的函数，相信你一定不会陌生：\n",
    "\n",
    "    >>> pi = 3.14\n",
    "    >>> def circle_area(r):\n",
    "    ...     return pi * r ** 2\n",
    "    ...\n",
    "    >>> circle_area(2)\n",
    "    12.56\n",
    "\n",
    "这段代码将圆面积计算功能组织成一个函数 circle_area ，圆半径 r 以参数形式作为输入，函数负责计算面积，并将结果作为返回值输出。这样一来，任何需要计算圆面积的地方，只需要调用 circle_area 即可，达到了功能分解以及代码复用的目的。\n",
    "\n",
    "我们知道 Python 中一切都是对象，函数也是一种对象。那么，作为一等对象的函数，到底长什么模样，有什么特殊行为呢？Python 代码又是如何一步步变身为函数对象的呢？洞悉函数秘密后，可以实现哪些有趣的功能呢？带着这些疑问，我们开始探索函数对象。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## 函数对象长啥样"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "首先，借助内建函数 dir 观察函数对象，发现了不少新属性：\n",
    "\n",
    "    >>> dir(circle_area)\n",
    "    ['__annotations__', '__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__get__', '__getattribute__', '__globals__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__kwdefaults__', '__le__', '__lt__', '__module__', '__name__', '__ne__', '__new__', '__qualname__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__']\n",
    "\n",
    "通过名字，我们可以猜测出某些字段的功能。 __code__ 应该是一个 代码对象 ，里面保存着函数的 字节码 。从字节码中，我们可以清晰地读懂函数的执行逻辑，跟 Python 语句表达的意思一模一样：\n",
    "\n",
    "    >>> circle_area.__code__\n",
    "    <code object circle_area at 0x10d52d270, file \"<stdin>\", line 1>\n",
    "    >>> import dis\n",
    "    >>> dis.dis(circle_area.__code__)\n",
    "      2           0 LOAD_GLOBAL              0 (pi)\n",
    "                  2 LOAD_FAST                0 (r)\n",
    "                  4 LOAD_CONST               1 (2)\n",
    "                  6 BINARY_POWER\n",
    "                  8 BINARY_MULTIPLY\n",
    "                 10 RETURN_VALUE\n",
    " \n",
    " 又如， __globals__ 顾名思义应该是函数的全局名字空间，全局变量 pi 的藏身之地。\n",
    "\n",
    "    >>> circle_area.__globals__\n",
    "    {'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__spec__': None, '__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, 'pi': 3.14, 'circle_area': <function circle_area at 0x10d573950>, 'dis': <module 'dis' from '/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/dis.py'>}\n",
    "\n",
    "确实如此， pi 的出现证实了我们的猜测。由于 pi 是在 __main__ 模块中定义的，保存在模块的属性空间内。那么， __globals__ 到底是模块属性空间本身，还是它的一个拷贝呢？我们接着观察：\n",
    "\n",
    "    >>> circle_area.__globals__ is sys.modules['__main__'].__dict__\n",
    "    True\n",
    "    \n",
    "注意到， sys.modules 保存着 Python 当前所有已导入的模块对象，包括 __main__ 模块。我们取出 __main__ 模块的属性空间 __dict__ 与函数全局名字空间对比，发现他们是同一个 dict 对象。原来函数全局名字空间和模块属性空间就是这样紧密绑定在一起的！\n",
    "\n",
    "此外，我们还可以找到函数名以及所属模块名这两个字段：\n",
    "\n",
    "    >>> circle_area.__name__\n",
    "    'circle_area'\n",
    "    >>> circle_area.__module__\n",
    "    '__main__'\n",
    "\n",
    "函数对象关键属性整理如下：\n",
    "\n",
    "|属性|描述|\n",
    "|:-:|:-:|\n",
    "|code|代码对象，包含函数字节码|\n",
    "|globals|函数全局名字空间|\n",
    "|module|函数所属模块名|\n",
    "|name|函数名字|\n",
    "\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_01.png\" width=800 height=600> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T05:53:50.153234Z",
     "start_time": "2021-03-29T05:53:50.149223Z"
    },
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 函数对象如何创建"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "我们已经初步看清 **函数** 对象的模样，它和 **代码** 对象关系密切，**全局名字空间** 就是它所在 **模块** 对象的 **属性空间**。那么，Python 又是如何完成从代码到函数对象的转变的呢？想了解这其中的秘密，还是得从字节码入手。\n",
    "\n",
    "现在我们要想方设法搞到定义函数的字节码，先将函数代码作为文本保存起来：\n",
    "\n",
    "    >>> text = '''\n",
    "    ... pi = 3.14\n",
    "    ... def circle_area(r):\n",
    "    ...     return pi * r ** 2\n",
    "    ... '''\n",
    "    \n",
    "然后，调用 compile 函数编译函数代码，得到一个 代码 对象：\n",
    "\n",
    "    >>> code = compile(text, 'test', 'exec')\n",
    "    \n",
    "根据 **虚拟机** 部分的学习，我们知道 **作用域** 与 **代码** 对象之间的一一对应的关系。定义函数的这段代码虽然简短，里面却包含了两个不同的 **作用域** ：一个是模块级别的 **全局作用域** ，一个函数内部的 **局部作用域** ：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_02.png\" width=800 height=600> \n",
    "\n",
    "那么，为啥 compile 函数只返回一个代码对象呢？因为局部代码对象作为一个 常量 ，藏身于全局代码对象中。而 compile 函数则只需返回全局代码对象：\n",
    "\n",
    "    >>> code.co_names\n",
    "    ('pi', 'circle_area')\n",
    "    >>> code.co_consts\n",
    "    (3.14, <code object circle_area at 0x10e179420, file \"test\", line 3>, 'circle_area', None)\n",
    "\n",
    "    >>> code.co_consts[1]\n",
    "    <code object circle_area at 0x10e179420, file \"test\", line 3>\n",
    "    >>> code.co_consts[1].co_names\n",
    "    ('pi',)\n",
    "    >>> code.co_consts[1].co_consts\n",
    "    (None, 2)\n",
    "    \n",
    "    \n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_03.png\" width=800 height=600> \n",
    "    \n",
    "看来，函数对象诞生的秘密就藏在 MAKE_FUNCTION 指令中。\n",
    "\n",
    "开始深入源码研究 MAKE_FUNCTION 指令前，我们先推演一遍虚拟机执行这段字节码的全过程。假设 circle_area 在 __main__ 模块中定义，全局代码对象则作为模块代码执行，以模块 **属性空间** 为 **全局名字空间** 和 **局部名字空间** 。前两行字节码与函数创建无关，在将 3.14 作为 pi 值保存到 **局部名字空间**，它也是模块的 **属性空间** ：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_04.png\" width=800 height=600> \n",
    "\n",
    "接下来两行字节码将两个常量加载到栈顶，为创建函数做最后的准备：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_05.png\" width=800 height=600> \n",
    "\n",
    "这两个常量是创建函数最重要的参数，一个指定函数的 **代码** 对象，一个指定 **函数名** 。 MAKE_FUNCTION 字节码从栈顶取出这两个参数，完成 **函数** 对象的创建，并将其放置于栈顶。此外，函数对象继承了当前 **帧** 对象的 **全局名字空间** 。因此， circle_area 不管在何处调用，其全局名字空间一定是就是它所在模块 ( __main__ ) 的 **属性空间** 。\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_06.png\" width=800 height=600> \n",
    "\n",
    "紧接着的 STORE_NAME 指令我们已经非常熟悉了，它将创建好的函数对象从栈顶弹出，并保存到局部名字空间。\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_07.png\" width=800 height=600> \n",
    "\n",
    "注意到，这个 **局部名字空间** 正好是模块对象的 **属性空间** ！如果函数是在模块 demo 中定义，则可以这样引用：\n",
    "\n",
    "    >>> import demo\n",
    "    >>> demo.circle_area(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### MAKE_FUNCTION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "经过 **虚拟机** 部分学习，我们对研究字节码的套路早已了然于胸。虚拟机处理字节码的逻辑位于 Python/ceval.c ：\n",
    "\n",
    "        TARGET(MAKE_FUNCTION) {\n",
    "            PyObject *qualname = POP();\n",
    "            PyObject *codeobj = POP();\n",
    "            PyFunctionObject *func = (PyFunctionObject *)\n",
    "                PyFunction_NewWithQualName(codeobj, f->f_globals, qualname);\n",
    "\n",
    "            Py_DECREF(codeobj);\n",
    "            Py_DECREF(qualname);\n",
    "            if (func == NULL) {\n",
    "                goto error;\n",
    "            }\n",
    "\n",
    "            if (oparg & 0x08) {\n",
    "                assert(PyTuple_CheckExact(TOP()));\n",
    "                func ->func_closure = POP();\n",
    "            }\n",
    "            if (oparg & 0x04) {\n",
    "                assert(PyDict_CheckExact(TOP()));\n",
    "                func->func_annotations = POP();\n",
    "            }\n",
    "            if (oparg & 0x02) {\n",
    "                assert(PyDict_CheckExact(TOP()));\n",
    "                func->func_kwdefaults = POP();\n",
    "            }\n",
    "            if (oparg & 0x01) {\n",
    "                assert(PyTuple_CheckExact(TOP()));\n",
    "                func->func_defaults = POP();\n",
    "            }\n",
    "\n",
    "            PUSH((PyObject *)func);\n",
    "            DISPATCH();\n",
    "        }\n",
    "        \n",
    "1. 第 2-3 行，从栈顶弹出关键参数；\n",
    "2. 第 4-5 行，调用 PyFunction_NewWithQualName 创建 函数 对象， 全局名字空间 来源于当前 帧 对象；\n",
    "3. 第 13-16 行，如果函数为 闭包函数 ，从栈顶取 闭包变量 ；\n",
    "4. 第 17-20 行，如果函数包含注解，从栈顶取注解；\n",
    "5. 第 21-28 行，如果函数参数由默认值，从栈顶取默认值，分为普通默认值以及非关键字默认值两种；\n",
    "\n",
    "PyFunction_NewWithQualName 函数在 Objects/funcobject.c 源文件中实现，主要参数有 3 个：\n",
    "\n",
    "- code，代码对象；\n",
    "- globals， 全局名字空间；\n",
    "- qualname，函数名；\n",
    "\n",
    "PyFunction_NewWithQualName 函数则实例化 **函数** 对象 ( PyFunctionObject )，并根据参数初始化相关字段。\n",
    "\n",
    "当然了，我们也可以用 Python 语言模拟这个过程。根据 **对象模型** 中规则，调用 **类型** 对象即可创建 **实例** 对象。只是 Python 并没有暴露 **函数类型** 对象，好在它不难找：\n",
    "\n",
    "    >>> def a():\n",
    "    ...     pass\n",
    "    ... \n",
    "    >>> function = a.__class__\n",
    "    >>> function\n",
    "    <class 'function'>\n",
    "    \n",
    "我们随便定义了一个函数，然后通过 __class__ 找到它的 类型 对象，即 函数类型 对象。\n",
    "\n",
    "然后，我们准备函数的 **代码** 对象：\n",
    "\n",
    "    >>> text = '''\n",
    "    ... def circle_area(r):\n",
    "    ...     return pi * r ** 2\n",
    "    ... '''\n",
    "    >>> code = compile(text, 'test', 'exec')\n",
    "    >>> func_code = code.co_consts[0]\n",
    "    >>> func_code\n",
    "    <code object circle_area at 0x10e029150, file \"test\", line 2>\n",
    "    \n",
    "由此一来，函数三要素便已俱备，调用 **函数类型** 对象即可完成临门一脚：\n",
    "\n",
    "    >>> circle_area = function(func_code, globals(), 'circle_area')\n",
    "    >>> circle_area\n",
    "    <function circle_area at 0x10e070620>\n",
    "    \n",
    "至此，我们得到了梦寐以求的 **函数** 对象，而且是以一种全新的方式！\n",
    "\n",
    "但是，我们把全局变量 pi 忘在脑后了，没有它函数跑不起来：\n",
    "\n",
    "    >>> circle_area(2)\n",
    "    Traceback (most recent call last):\n",
    "      File \"<stdin>\", line 1, in <module>\n",
    "      File \"test\", line 3, in circle_area\n",
    "    NameError: name 'pi' is not defined\n",
    "    \n",
    "这难不倒我们，加上便是：\n",
    "\n",
    "    >>> pi = 3.14\n",
    "    >>> circle_area(2)\n",
    "    12.56\n",
    "    \n",
    "    >>> circle_area.__defaults__ = (1,)\n",
    "    \n",
    " 由此一来，如果调用 circle_area 函数时未指定参数，则默认以 1 为参数。\n",
    "\n",
    "    >>> circle_area()\n",
    "    3.14\n",
    "    >>> circle_area(3)\n",
    "    28.26\n",
    "    >>> circle_area(1)\n",
    "    3.14"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## 函数调用与虚拟机软件栈"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "我们已经掌握了创建函数对象的秘密，并发挥自己的聪明才智以一种全新的方式创造了一个函数对象。虽然在实际项目中，我们不会这么做，但这种新尝试让我们可以更好地理解函数的行为。\n",
    "\n",
    "现在，我们又对函数调用的秘密充满好奇。函数是怎么调用的？参数和返回值是如何调用的？递归又是如何实现的？带着这些问题，我们再次启程，研究 circle_area 这个我们既熟悉又陌生的函数。\n",
    "\n",
    "我们将 circle_area 定义在 geometry 模块中，文件名为 geometry.py ：\n",
    "    \n",
    "    pi = 3.14\n",
    "\n",
    "    def circle_area(r):\n",
    "        return pi * r ** 2\n",
    "\n",
    "    def cylinder_volume(r, h):\n",
    "        return circle_area(r) * h\n",
    "        \n",
    "注意到，模块中还有另一个函数 cylinder_volume 用于计算圆柱体体积，参数 r 是底面圆的半径，参数 h 是圆柱体高度。 cylinder_volume 先调用 circle_area 计算底面面积，再乘以高度得到圆柱体积。\n",
    "\n",
    "进入 geometry.py 所在目录，并启动 Python 终端，将 geometry.py 模块导入，即可调用相关函数：\n",
    "\n",
    "    >>> from geometry import circle_area, cylinder_volume\n",
    "    >>> circle_area(1.5)\n",
    "    7.065\n",
    "    \n",
    "如果你不想进入 geometry.py 所在目录，也可以将其路径加入到 sys.path ，这个方法我们在模块机制中介绍过：\n",
    "\n",
    "    >>> import sys\n",
    "    >>> sys.path.append('/some/path')\n",
    "    \n",
    "开始讨论函数调用流程之前，我们先来看看从 geometry 模块导入相关函数后虚拟机内部的状态：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_08.png\" width=800 height=600> \n",
    "\n",
    "- __main__ 模块是 Python 启动后的执行入口，每个 Python 程序均从 __main__ 开始执行；\n",
    "- geometry 是我们导入的模块，它有一个 __dict__ 属性，指向模块属性空间；\n",
    "- geometry 初始化后，属性空间里有一个浮点属性 pi 以及两个函数对象， circle_area 和 cylinder_colume ；\n",
    "- 两个函数的 **全局名字空间** 与模块对象的 **属性空间** 是同一个 dict 对象；\n",
    "- 两个函数都有一个 **代码对象** ，保存函数 **字节码** 以及 **名字** 、 **常量** 等静态上下文信息；\n",
    "- 往下阅读前请务必理解该状态图，有疑问请复习虚拟机模块机制以及函数创建等章节，以加深理解；\n",
    "\n",
    "每个 Python 程序都有一个 __main__ 模块，以及与 __main__ 模块对应的 **栈帧** 对象。__main__ 模块是 Python 程序的入口，而与其对应的栈帧对象则是整个程序调用栈的起点。\n",
    "\n",
    "当我们在交互式终端输入语句时，也是类似的。 Python 先将代码编译成代码对象，再创建一个 **栈帧** 对象执行该代码对象。以 circle_area(1.5) 为例，编译可得到这样的字节码：\n",
    "\n",
    "      1           0 LOAD_NAME                0 (circle_area)\n",
    "                  2 LOAD_CONST               0 (1.5)\n",
    "                  4 CALL_FUNCTION            1\n",
    "                  6 PRINT_EXPR\n",
    "                  8 LOAD_CONST               1 (None)\n",
    "                 10 RETURN_VALUE\n",
    " 随后，Python 创建栈帧对象作为执行环境，准备执行编译后的代码对象：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_09.png\" width=800 height=600> \n",
    "\n",
    "注意到，栈帧对象全局名字空间、局部名字空间均指向 __main__ 模块的属性空间。 circle_area(1.5) 的语句中，有些我们已经非常熟悉了。第一条字节码，将名为 circle_area 的对象，加载到栈顶，这是我们导入的函数。第二条字节码，将常量 1.5 加载到栈顶，这是准备传递给函数的变量。执行这两个字节码后，虚拟机状态变为：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_10.png\" width=800 height=600> \n",
    "\n",
    "接着是 CALL_FUNCTION 字节码，顾名思义，我们知道正式它完成了调动函数的使命。CALL_FUNCTION 字节码的处理逻辑同样位于 Python/ceval.c 这个文件，有兴趣的童鞋可以阅读一下源码，这里用通俗的语言结合图示讲解这个字节码的作用。\n",
    "\n",
    "CALL_FUNCTION 先创建一个新栈帧对象，作为 circle_name 函数的执行环境。新栈帧对象通过 f_back 指针，指向前一个栈帧对象，形成一个调用链。栈帧对象从函数对象取得 **代码** 对象，以及执行函数时的全局名字空间：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_11.png\" width=800 height=600> \n",
    "\n",
    "此外，注意到执行函数的栈帧对象 f_locals 字段为空，而不是跟 f_globals 一样执行一个 dict 对象。由于函数有多少局部变量是固定的，代码编译时就能确定。因此，没有必要用字典来实现局部名字空间，只需把局部变量依次编号，保存在栈底即可 ( r=1.5 处)。这样一来，通过编号即可快速存取局部变量，效率比字典更高。于此对应，有一个特殊的字节码 LOAD_FAST 用于加载局部变量，以操作数的编号为操作数。\n",
    "\n",
    "circle_area 的字节码我们已经很熟悉了，便不再赘述了，请动手在栈帧上推演一番。最后， RETURN_VALUE 字节码将结算结果返回给调用者，执行权现在交回调用者的 CALL_FUNCTION 字节码。CALL_FUNCTION 先将结果保存到栈顶并着手回收 circle_area 函数的栈帧对象。\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_12.png\" width=800 height=600> \n",
    "\n",
    "嵌套调用也是类似的，以 cylinder_volume(1.5, 2) 为例：\n",
    "\n",
    "    >>> cylinder_volume(1.5, 2)\n",
    "    14.13\n",
    "    \n",
    "Python 交互式终端同样先对这个语句进行编译，得到这样的字节码：\n",
    "\n",
    "      1           0 LOAD_NAME                0 (cylinder_volume)\n",
    "                  2 LOAD_CONST               0 (1.5)\n",
    "                  4 LOAD_CONST               1 (2)\n",
    "                  6 CALL_FUNCTION            2\n",
    "                  8 PRINT_EXPR\n",
    "                 10 LOAD_CONST               2 (None)\n",
    "                 12 RETURN_VALUE\n",
    "                 \n",
    " 然后， Python 虚拟机以 __main__ 栈帧对象为环境，执行这段字节码。当虚拟机执行到 CALL_FUNCTION 这个字节码时，创建新栈帧对象，准备执行函数调用。初始新栈帧对象时，函数参数来源于当前栈顶，而全局名字空间与代码对象来源于被调用函数对象。新栈帧对象初始化完毕，虚拟机便跳到新栈帧，开始执行 cylinder_volume 的字节码。cylinder_volume 字节码中也有 CALL_FUNCTION 指令，调用 circle_area 函数。虚拟机依样画葫芦，为 circle_area 准备栈帧，并开始执行 circle_area 的字节码：\n",
    " \n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_13.png\" width=800 height=600> \n",
    " \n",
    " 这样一来，随着函数调用的深入，栈帧链逐渐伸长；随着函数执行完毕并返回，栈帧链逐渐收缩。维护栈帧链条的关键是栈帧对象的 f_back 指针，它总是指向上个一栈帧对象，也就是调用者的栈帧，如上图红色箭头。我们在调试程序时，可以查看完整的堆栈信息，也是 f_back 指针的功劳。\n",
    "\n",
    "正常情况下，函数调用层数不会太深，但递归调用就说不定了。我们来看一个典型的递归例子，斐波那契数列计算：\n",
    "\n",
    "    def fibonacci(n):\n",
    "        if n == 0:\n",
    "            return 0\n",
    "        if n == 1:\n",
    "            return 1\n",
    "\n",
    "        return fibonacci(n-1) + fibonacci(n-2）\n",
    "\n",
    "以 fibonacci(5) 为例，需要递归调用 fibonacii(4) ，而 fibonacii(4) 需要调用 fibonacci(3) ，以此类推。递归调用一直向下延伸，直到 fibonacci(1) 。因此，调用链最长时是这样子的：\n",
    "\n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_14.png\" width=800 height=600> \n",
    "\n",
    "由此可见，调用 fibonacci(10000) 时，栈帧链长度将达到 10000 。因此，实现递归算法需要特别小心，不免栈内存溢出。此外，每次函数调用都需要创建并销毁栈帧对象，是否意味着性能低下呢？\n",
    "\n",
    "这是肯定的。Python 内部通过 **内存池** 技术对频繁分配回收内存的场景进行了优化，栈帧频繁创建回收导致的性能问题得到一定的缓解。内存池技术将在 Python **内存管理** 部分进行介绍，敬请期待。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## 面试必问：嵌套函数、闭包与装饰器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Python 函数可以嵌套定义，我们先考察一个典型的例子：\n",
    "\n",
    "    def adder(n):\n",
    "        def handler(x):\n",
    "            return n+x\n",
    "        return handler\n",
    "\n",
    "adder 函数负责创建处理函数 handler ，处理函数计算参数 x 与固定值 n 的和。因此，如果我们需要一个为参数加上 1 的函数，调用 adder(1) 即可轻松得到：\n",
    "\n",
    "    >>> add1 = adder(1)\n",
    "    >>> add1(10)\n",
    "    11\n",
    "    >>> add1(15)\n",
    "    16\n",
    "    \n",
    "同样，如果我们需要一个将参数加上 5 的函数，调用 adder(5) 即可：\n",
    "\n",
    "    >>> add5 = adder(5)\n",
    "    >>> add5(10)\n",
    "    15\n",
    "    >>> add5(15)\n",
    "    20\n",
    "\n",
    "很显然，对于 add1 来说，n 的值是 1 ；对于 add5 来说，n 的值是 5 ；两者保存独立，互不干扰。\n",
    "\n",
    "理论上，当函数 adder 返回，局部变量 n 应该就被回收了，为什么 handler 函数还能访问到它呢？\n",
    "\n",
    "另外，像 adder 函数和 handler 函数这种嵌套写法，到底有什么作用？适用于什么开发场景？有什么需要特别注意的地方吗？\n",
    "\n",
    "为了解答这诸多疑惑，我们需要深入学习**嵌套函数** 与 **闭包** 变量的来龙去脉。\n",
    "\n",
    "**嵌套函数**\n",
    "\n",
    "像 adder 函数和 handler 这样，在一个函数的函数体内定义另一个函数，就构成了 **嵌套函数** 。我们在 **虚拟机** 部分讲解 **作用域** 时，已对嵌套函数有所了解。你还记得嵌套函数与作用域之间密切的联系吗？\n",
    "\n",
    "根据我们在虚拟机部分学到的知识，adder-handler 这段简单的代码却包含着 3 个不同的作用域：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_15.png\" width=800 height=600> \n",
    "\n",
    "作用域是一个静态概念，由 Python 代码语法决定，与编译后产生的 **代码对象** 一一对应。作用域规定了能够被某个代码块访问的变量有哪些，但对变量具体的值则一概不关心。\n",
    "\n",
    "一旦 Python 程序开始运行，虚拟机需要为作用域中的变量分配一定的存储空间，这就是 **名字空间** 。名字空间依照作用域规则实现，它决定了某个变量在运行时的取值，可以看做是作用域在运行时的动态表现方式。\n",
    "\n",
    "当 adder 函数执行时，作用域 A 在虚拟机中表现为 **全局** 名字空间，作用域 B 表现为 **局部** 名字空间：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_16.png\" width=800 height=600> \n",
    "\n",
    "当 handler 函数执行时，例如调用 adder(10) 时，作用域 A 在虚拟机中表现为 **全局** 名字空间，作用域 B 表现为 **闭包** 名字空间：作用域 C 表现为 **局部** 名字空间：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_17.png\" width=800 height=600> \n",
    "\n",
    "全局与局部这两个名字空间我们已经非常熟悉了，那么 **闭包** 名字空间又该如何理解呢？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 闭包"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**闭包** ( closure ) 是 **词法闭包** ( Lexical Closure ) 的简称，是指引用了自由变量的函数。这些被引用的自由变量将和这个函数一同存在，即使已经离开了创造它的环境也不例外。\n",
    "\n",
    "以 adder(10) 为例，它是一个 handler 函数对象，闭包变量 n 值总是 10 。那么，内层函数是如何访问闭包作用域的呢？我们对函数代码对象进行反汇编，从中可以看出端倪：\n",
    "\n",
    "    >>> add10 = adder(10)\n",
    "    >>> add10\n",
    "    <function adder.<locals>.handler at 0x10dc2b6a8>\n",
    "    >>> add10.__code__\n",
    "    <code object handler at 0x10dbe5150, file \"<stdin>\", line 2>\n",
    "    >>> dis.dis(add10.__code__)\n",
    "      3           0 LOAD_DEREF               0 (n)\n",
    "                  2 LOAD_FAST                0 (x)\n",
    "                  4 BINARY_ADD\n",
    "                  6 RETURN_VALUE\n",
    "  \n",
    "我们发现一条全新的字节码 LOAD_DEREF ，正是它执行了闭包变量查找工作！LOAD_FAST 指令则负责局部变量查找，而局部名字空间我们已经有所了解，它藏身于栈帧对象中。\n",
    "\n",
    "那么，闭包名字空间又藏在哪呢？\n",
    "\n",
    "顺着虚拟机处理 LOAD_DEREF 字节码的代码，不难找到答案。与该字节码处理相关的源码位于 Python/ceval.c 文件，有兴趣的同学可以深入看一下。\n",
    "\n",
    "因篇幅所限，这里直接给出答案：当闭包函数 handler 执行时，闭包变量藏身于 PyFrameObject 中。\n",
    "\n",
    "还记得吗？每次函数调用虚拟机都会创建一个 PyFrameObject 对象，用于保存函数调用上下文。全局名字空间与局部名字空间都藏身其中，闭包名字空间也不例外。\n",
    "\n",
    "前面章节提过，PyFrameObject 结构体最后部分是不固定的，依次存放着静态局部名字空间、闭包名字空间以及临时栈。以 add10(1) 为例，函数运行时 PyFrameObject 状态如下如下：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_18.png\" width=800 height=600> \n",
    "\n",
    "我们重点关注 PyFrameObject 末尾处，局部变量、闭包变量以及临时栈依次排列。\n",
    "\n",
    "由于函数局部变量、闭包变量个数在编译阶段就能确定，运行时并不会增减，因此无须用 dict 对象来保存。相反，将这些变量依次排列保存在数组中，然后通过数组下标来访问即可。这就是所谓的静态名字空间。\n",
    "\n",
    "对于局部变量 n ，数组对应的槽位保存着整数对象 1 的地址，表示 n 与 1 绑定。而闭包变量 x 则略有差别，槽位不直接保存整数对象 10 ，而是通过一个 PyCellObject 间接与整数对象 10 绑定。\n",
    "\n",
    "那么，闭包变量是如何完成初始化的呢？\n",
    "\n",
    "我们知道全局名字空间 f_globals 来源于函数对象，那么闭包名字空间是否也是如此呢？再次研究 CALL_FUNCTION 字节码处理逻辑，印证了我们的猜测：\n",
    "\n",
    "函数对象 PyFunctionObject 中有一个字段 func_closure ，保存着函数所有闭包变量。我们可以通过名字 __closure__ 可以访问到这个底层结构体字段：\n",
    "\n",
    "    >>> add10.__closure__\n",
    "    (<cell at 0x10dc09e28: int object at 0x10da161a0>,)\n",
    "    \n",
    "这是一个由 PyCellObject 组成的元组，PyCellObject 则保存着闭包变量的值。当函数调用发生时，Python 虚拟机创建 PyFrameObject 对象，并从函数对象取出该元组，依次填充相关静态槽位。\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_19.png\" width=800 height=600> \n",
    "\n",
    "接着我们来考察 PyCellObject 对象的行为，通过 cell_contents 属性可以访问闭包变量值：\n",
    "\n",
    "    >>> add10.__closure__[0]\n",
    "    <cell at 0x10dc09e28: int object at 0x10da161a0>\n",
    "    >>> add10.__closure__[0].cell_contents\n",
    "    10\n",
    "    \n",
    "我们尝试将函数闭包变量进行修改，发现函数的行为将受到影响：\n",
    "\n",
    "    >>> add10.__closure__[0].cell_contents = 20\n",
    "    >>> add10(1)\n",
    "    21\n",
    "    \n",
    "这种行为虽合乎常理，但千万不要在实际项目中应用，不然可能会踩坑！\n",
    "\n",
    "至此，一切都明了了！从 adder 函数字节码告诉我们，它将 handler 函数所有闭包变量存为 tuple 对象，再执行 MAKE_FUNCTION 字节码完成函数对象创建。该元组最终将作为 closure 参数传给函数对象，并保存在 func_closure 字段中。\n",
    "\n",
    "    >>> dis.dis(adder)\n",
    "      2           0 LOAD_CLOSURE             0 (n)\n",
    "                  2 BUILD_TUPLE              1\n",
    "                  4 LOAD_CONST               1 (<code object handler at 0x102864150, file \"<stdin>\", line 2>)\n",
    "                  6 LOAD_CONST               2 ('adder.<locals>.handler')\n",
    "                  8 MAKE_FUNCTION            8\n",
    "                 10 STORE_FAST               1 (handler)\n",
    "\n",
    "      4          12 LOAD_FAST                1 (handler)\n",
    "                 14 RETURN_VALUE\n",
    "                 \n",
    " 那么，为什么闭包变量要通过 PyCellObject 间接引用呢？我们将函数对象与运行时的栈帧对象放在一起来讨论：\n",
    " \n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_20.png\" width=800 height=600> \n",
    "\n",
    "最新的 Python 提供了 nonlocal 关键字，支持修改闭包变量。如果没有 PyCellObject ，函数在运行时直接修改 PyFrameObject ，函数返回就被回收了。借助 PyCellObject ，函数在运行时修改的是 ob_ref 。这样一来，就算函数返回，修改还是随函数而存在。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 装饰器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "借助闭包，我们可以让函数具备搭积木的魔法，例如：函数调用前后做一些统一的处理操作。\n",
    "\n",
    "事不宜迟，我们来实践一下，实现 log_call 函数，在指定函数调用前后各输出一句话：\n",
    "\n",
    "    def log_call(func):\n",
    "        def handler(*args, **kwargs):\n",
    "            print('start to call')\n",
    "            result = func(*args, **kwargs)\n",
    "            print('end of call')\n",
    "            return result\n",
    "        return handler\n",
    "        \n",
    "log_call 接受一个函数对象 func 作为参数，这就是我们想为其注入魔法，以便在调用前后输出提示的函数，姑且称为 **原函数** 。它返回一个闭包函数 handler ，姑且称为 **代理函数** ，闭包变量 func 与 handler 紧紧绑定在一起。\n",
    "\n",
    "当 handler 函数被调用时，它先输出提示，再调用原函数 func ，函数调用完毕再次输出提示。注意到，由于原函数参数是未知的，因此 handler 通过可变参数 *args 以及 **kwargs 进行传递。\n",
    "\n",
    "现在，我们将 log_call 用起来！对于任意的函数 foo ，log_call 轻松即可为它注入魔法：\n",
    "\n",
    "    >>> def foo():\n",
    "    ...     print('foo processing')\n",
    "    ...\n",
    "    >>> log_call(foo)()\n",
    "    start to call\n",
    "    foo processing\n",
    "    end of call\n",
    "    \n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_21.png\" width=800 height=600> \n",
    " \n",
    " 如果我想为其他函数，例如 bar 注入 log_call 魔法，可以这样实现：\n",
    "\n",
    "    def bar():\n",
    "        print('bar processing')\n",
    "\n",
    "    bar = log_call(bar)\n",
    "\n",
    "    bar()\n",
    "    \n",
    "这代码未免太丑陋了！为此，Python 引入了语法糖 @xxxx ：\n",
    "\n",
    "    @log_call\n",
    "    def bar():\n",
    "        print('bar processing')\n",
    "\n",
    "    bar()\n",
    "    \n",
    "这段代码与上面那段完全等价，却更加优雅、清晰！\n",
    "\n",
    "像 log_call 这样，为其他函数注入新功能的函数，就是所谓的 **装饰器** ( decorator )。因篇幅所限，我们仅吃了一味开胃菜，诸多细节来不及展开，后续章节将进一步深入讨论。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## 高阶函数与函数式编程"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "从前面章节，我们知道 Python 函数是以对象的形式实现的，属于 **一等对象** ( first-class object )。根据编程语言理论，一等对象必须满足以下条件：\n",
    "\n",
    "- 可在运行时创建；\n",
    "- 能赋值给变量或者某种数据结构；\n",
    "- 能作为参数传递给函数；\n",
    "- 能作为函数执行结果返回；\n",
    "\n",
    "Python 函数同时满足这几个条件，因而也被称为 **一等函数** 。 **高阶函数** 则是指那些以函数为参数，或者将函数作为结果返回的函数。对高阶函数稍加利用，便能玩出很多花样来。本节从一些典型的案例入手，讲解 Python 函数高级用法。合理应用函数式编程技巧，不仅能让代码更加简洁优雅，还能提高开发效率和程序质量。\n",
    "\n",
    "函数式编程技巧最适合用在数据处理场景，接下来以成绩单计算为例，展开讲解。原始数据如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T07:55:27.166305Z",
     "start_time": "2021-03-29T07:55:27.153298Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "scores = [\n",
    "    {\n",
    "        'name': '小雪',\n",
    "        'chinese': 90,\n",
    "        'math': 75,\n",
    "        'english': 85,\n",
    "    },\n",
    "    {\n",
    "        'name': '小明',\n",
    "        'chinese': 70,\n",
    "        'math': 95,\n",
    "        'english': 80,\n",
    "    },\n",
    "    {\n",
    "        'name': '小丽',\n",
    "        'chinese': 85,\n",
    "        'math': 85,\n",
    "        'english': 90,\n",
    "    },\n",
    "    {\n",
    "        'name': '小宇',\n",
    "        'chinese': 85,\n",
    "        'math': 95,\n",
    "        'english': 90,\n",
    "    },\n",
    "    {\n",
    "        'name': '小刚',\n",
    "        'chinese': 65,\n",
    "        'math': 70,\n",
    "        'english': 55,\n",
    "    },\n",
    "    {\n",
    "        'name': '小新',\n",
    "        'chinese': 85,\n",
    "        'math': 85,\n",
    "        'english': 80,\n",
    "    },\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### sorted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "排序是我们再熟悉不过的场景，如果待排序元素可以直接比较，调用 sorted 函数即可："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T07:55:47.640143Z",
     "start_time": "2021-03-29T07:55:47.622154Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    ">>> numbers = [2, 8, 6, 9, 7, 0, 1, 7, 0, 3]\n",
    ">>> sorted(numbers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "对比较复杂的数据进行排序，则需要一些额外的工作。假如语文老师想对语文成绩进行排序，改如何进行呢？\n",
    "\n",
    "sorted 支持指定一个自定义排序函数 key ，该函数以列表元素为参数，返回一个值决定该元素的次序。由于我们需要根据语文成绩对元素进行排序，因此需要实现一个函数将语文成绩提取出来作为比较基准："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T07:56:36.468004Z",
     "start_time": "2021-03-29T07:56:36.464003Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def by_chinese(item):\n",
    "    return item['chinese']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "现在只需要将 by_chinese 函数作为 key 参数传给 sorted 即可实现语文成绩排序："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T07:56:38.002366Z",
     "start_time": "2021-03-29T07:56:37.986376Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "for item in sorted(scores, key=by_chinese):\n",
    "    print(item['name'], item['chinese'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "自定义排序函数还可以控制升降序，如果需要按分数从高到底依次排序，可以返回成绩的负数作为排序基准："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T07:56:52.028923Z",
     "start_time": "2021-03-29T07:56:52.022927Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def by_chinese_desc(item):\n",
    "    return -item['chinese']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T07:57:00.704213Z",
     "start_time": "2021-03-29T07:57:00.691219Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "for item in sorted(scores, key=by_chinese_desc):\n",
    "    print(item['name'], item['chinese'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "当然了，通过 sorted 函数 reverse 参数控制升降序，是一个更好的编程习惯，逻辑更清晰："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T07:57:27.615990Z",
     "start_time": "2021-03-29T07:57:27.609007Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "for item in sorted(scores, key=by_chinese, reverse=True):\n",
    "    print(item['name'], item['chinese'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### lambda"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "像 by_chinese 这样直接返回结果的极简函数，其实没有必要大动干戈，用 匿名函数 定义即可。Python 中的 lambda 关键字用于定义匿名函数，匿名函数只需给出参数列表以及一个表达式作为函数返回值：\n",
    "\n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_22.png\" width=800 height=600> \n",
    " \n",
    " 这样一来，by_chinese 这个自定义排序函数，可以这样来定义："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T07:59:13.152408Z",
     "start_time": "2021-03-29T07:59:13.147411Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "by_chinese = lambda item: item['chinese']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "相应地，我们实现语文成绩排序的代码编程这样子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T07:59:16.853499Z",
     "start_time": "2021-03-29T07:59:16.844490Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "for item in sorted(scores, key=lambda item: item['chinese']):\n",
    "    print(item['name'], item['chinese'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "数学老师来了，也只需要改动一点点，就能实现数学成绩排序了："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T07:59:43.655974Z",
     "start_time": "2021-03-29T07:59:43.643967Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "for item in sorted(scores, key=lambda item: item['math']):\n",
    "    print(item['name'], item['math'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "函数式编程语言一般都会提供 map 、 filter 以及 reduce 这 3 个高阶函数，再复杂的数据统计处理任务都可以转换成这些算子的组合。因此，不少大数据平台，例如 Hadoop 等，都以 map 、 reduce 为基础算子。Python 内部也自带了这几个高阶函数，我们分别来看："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "map 函数接受 转换函数 以及一个 可迭代对象 作为参数，返回另一个生成器，其元素是输入元素的转换结果：\n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_23.png\" width=800 height=600> \n",
    " \n",
    "班主任需要知道每位童鞋的平均分，map 函数能否帮到他？我们可以先写一个简单的函数来计算平均成绩："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T08:00:56.506305Z",
     "start_time": "2021-03-29T08:00:56.495316Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def calculate_total(item):\n",
    "    return {\n",
    "        'name': item['name'],\n",
    "        'total': item['chinese'] + item['math'] + item['english'],\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "这样一来，我们用 map 函数将成绩列表中的每个元素转换成平均成绩不就可以了吗？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T08:01:08.575335Z",
     "start_time": "2021-03-29T08:01:08.565335Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    ">>> for item in map(calculate_total, scores):\n",
    "...     print(item['name'], item['total'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "不仅如此，我们还可以进一步组合，对平均成绩进行排序："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T08:01:52.946111Z",
     "start_time": "2021-03-29T08:01:52.928097Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    ">>> for item in sorted(map(calculate_total, scores), key=lambda item: item['total'], reverse=True):\n",
    "...     print(item['name'], item['total'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "这就是高阶函数与算子组合的巨大威力，寥寥几行代码便完成了复杂的数据处理。尽管如此，这种风格也不能过分滥用。刻意应用高阶函数和函数式编程，代码可能更不可读，适得其反。\n",
    "\n",
    "此外，Python 程序一般不直接使用 map 函数，而是通过更有 Python 格调的推导式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T08:02:09.913533Z",
     "start_time": "2021-03-29T08:02:09.906536Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "totals = [calculate_total(item) for item in scores]\n",
    "totals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### filter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "filter 函数根据指定 **判定函数** ，对可迭代对象中的元素进行过滤，过滤结果同样以可迭代对象的形式返回。判定函数以某个元素为参数，返回 true 或 false 标识该元素是否应该出现在结果中。\n",
    "\n",
    "学校举办英语竞赛，英语老师想挑一些成绩比较好的童鞋前去参加。她设定了一个标准，上次考试成绩在 90 分或者以上。为了从成绩单中筛选出符合条件的童鞋，我们需要先定义一个判定函数：\n",
    "\n",
    "    lambda item: item['english'] >= 90\n",
    "    \n",
    "这个 lambda 函数某位同学成绩为输出，输出英语成绩是否大于等于 90 分。有了判定函数，调用 filter 即可将符合条件的同学给过滤出来："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T08:02:57.589848Z",
     "start_time": "2021-03-29T08:02:57.579868Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    ">>> for item in filter(lambda item: item['english'] >= 90, scores):\n",
    "...     print(item['name'], item['english'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "同样，Python 程序一般不直接使用 filter 函数，而是通过更有 Python 格调的推导式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T08:03:22.809611Z",
     "start_time": "2021-03-29T08:03:22.799598Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "candidates = [item['name'] for item in scores if item['english'] >= 90]\n",
    "candidates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### reduce"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "reduce 函数对可迭代对象的全部元素进行归并，归并方法由归并函数确定。归并函数以两个元素为输入，将两个元素进行合并，最后将结果作为输出返回。求和是一种典型的合并算子，归并函数如下"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T08:04:50.996301Z",
     "start_time": "2021-03-29T08:04:50.992303Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "add = lambda a, b: a + b\n",
    "\n",
    "add(1, 2),add(3, 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "将数学运算做成算子后，可以任何组合做一些有趣的事情。例如，与 reduce 函数组合，对整数序列求和："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T08:05:04.960756Z",
     "start_time": "2021-03-29T08:05:04.951761Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    ">>> numbers = [2, 8, 6, 9, 7, 0, 1, 7, 0, 3]\n",
    ">>> import functools\n",
    ">>> functools.reduce(add, numbers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "你也许会说，对序列求和，sum 函数就可以胜任了，何必搞得这么复杂？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T08:05:29.053466Z",
     "start_time": "2021-03-29T08:05:29.038460Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    ">>> sum(numbers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "是的，由于求和是一个很常见的操作，sum 函数被单独拿出来实现。reduce 作为通用的归并操作，则更灵活，威力也更强大。只需编写归并函数，便可实现任何你想要的归并操作。例如，对数字序列求乘积："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T08:05:40.018634Z",
     "start_time": "2021-03-29T08:05:40.013636Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    ">>> numbers = [6, 1, 2, 4, 3, 1, 3, 7, 5, 2]\n",
    ">>> functools.reduce(lambda a, b: a*b, numbers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### operator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Python 内置了常用数学运算算子，诸如加、减、乘、除等等，无须重复编写 lambda 函数来实现，以加法为例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T08:06:15.773252Z",
     "start_time": "2021-03-29T08:06:15.757266Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    ">>> import operator\n",
    ">>> operator.add(1, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "这样一来，对整数序列求乘积可以进一步简化成这样："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T08:06:27.218016Z",
     "start_time": "2021-03-29T08:06:27.202025Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    ">>> numbers = [6, 1, 2, 4, 3, 1, 3, 7, 5, 2]\n",
    ">>> functools.reduce(operator.mul, numbers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "operator 模块中的算子就不一一赘述了，请自行前去探索一番。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### functools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "functools 是一个用于操作高阶函数以及可调用对象的模块，里面提供了几个有意思的工具函数，非常有用。我们前面刚接触过的 reduce 函数，就是来自 functools 模块。除此之外，还有以下这些：\n",
    "\n",
    "- cached_property ，带缓存功能的属性装饰器，将留到类机制部分讲解；\n",
    "- lru_cache ，为函数提供缓存功能的装饰器；\n",
    "- partial ，生成 偏函数 ；\n",
    "- wraps ，包装函数，使它看起来更像另一个函数，一般在装饰器实现时用到；\n",
    "- etc\n",
    "\n",
    "接下来，我们以 partial 为例展开讲解，看看什么是偏函数以及如何在实际中应用偏函数。\n",
    "\n",
    "send_email 是一个调用 smtplib 发送邮件的函数，实现了邮件封装发送逻辑。SMTP 协议连接可以是普通可能是 TCP ，也可能是 SSL 连接。因此，smtplib 提供了两个不同的连接类，SMTP 以及 SMTP_SSL 。为了保持灵活性，send_email 函数也将连接对象参数化：\n",
    "\n",
    "    def send_email(host, port, user, password, fr, to, subject, body,\n",
    "            smtp_cls=smtplib.SMTP_SSL):\n",
    "        # ...\n",
    "\n",
    "        conn = smtp_cls(host, port)\n",
    "\n",
    "        # .\n",
    "        \n",
    "如果采用普通 TCP 连接，可以这样调用 send_email ：\n",
    "\n",
    "    send_email(\n",
    "        host=host,\n",
    "        port=port,\n",
    "        #...\n",
    "        smtp_cls=smtplib.SMTP,\n",
    "    )\n",
    "    \n",
    "如果采用普通 SSL 连接，由于 smtp_cls 参数默认就是 smtplib.SMTP_SSL ，因此可以这样调用 send_email ：\n",
    "\n",
    "    send_email(\n",
    "        host=host,\n",
    "        port=port,\n",
    "        #...\n",
    "    )\n",
    "    \n",
    "在某些场景，我们需要给 smtplib.SMTP_SSL 指定证书，例如：\n",
    "\n",
    "    smtplib.SMTP_SSL(\n",
    "        host=host,\n",
    "        port=port,\n",
    "        #...\n",
    "        keyfile='xxxx',\n",
    "        certfile='xxxx',\n",
    "    )\n",
    "    \n",
    "这是否意味着我们需要改造 send_email 函数呢？是否有办法将 smtplib.SMTP_SSL 与两个与证书相关的参数绑定后再传给 send_email 函数呢？当然了，这对 functools.partial 来说，完全不在话下：\n",
    "\n",
    "    send_email(\n",
    "    host=host,\n",
    "    port=port,\n",
    "    # ...\n",
    "    smtp_cls=functools.partial(smtplib.SMTP_SSL, keyfile='xxxx', certfile='xxxx'),\n",
    "    )\n",
    "    \n",
    "functools.partial 函数返回一个可调用对象，被调用时相当于调用 smtplib.SMTP_SSL ，而且自动附上那两个用于指定证书的参数。这样一来，当语句 smtp_cls(host, port) 执行时，最终等价于：\n",
    "\n",
    "    smtplib.SMTP_SSL(host, port, keyfile='xxxx', certfile='xxxx')\n",
    "    \n",
    "这就是偏函数典型的应用场景，你可能对 functools.partial 函数很好奇，后续我们找机会研究一番。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## 高级技巧：令人叹为观止的装饰器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "在 Python 中， **装饰器** 一般用来修饰函数，实现公共功能，达到代码复用的目的。 在函数定义前加上 @xxxx ，然后函数就注入了某些功能，很神奇！ 然而，这只是 **语法糖** 而已，起决定性作用的其实是前面介绍的 **闭包** 。\n",
    "\n",
    "本文从最简单的场景入手，带你认识装饰器的前世今生。在洞悉装饰器运行机制的基础上，我们一起研究装饰器的若干高级技巧，你将大开眼界——原来代码可以这么写！掌握这些高级技巧之后，你将进一步强化代码抽象能力，用更精简的代码完成更复杂的处理逻辑，大幅提升开发效率。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 场景"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "假设我们有一些工作函数，用来对数据进行不同的处理：\n",
    "\n",
    "    def work_bar(data):\n",
    "        pass\n",
    "\n",
    "    def work_foo(data):\n",
    "        pass\n",
    "        \n",
    "如果我们想在函数调用前后输出日志，该怎么办？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 古典工艺"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "最直接的做法，是在函数调用前后加上打日志的语句：\n",
    "\n",
    "    logging.info('begin call work_bar')\n",
    "    work_bar(1)\n",
    "    logging.info('call work_bar done')\n",
    "    \n",
    "试想，如果程序中有多处 work_bar 函数调用会怎样呢？想想就怕！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 函数封装"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "古典工艺比较原始，每次函数调用都要写一遍 logging ，引入太多代码冗余。我们知道，函数是程序逻辑封装和代码复用的基本单位。借助函数，我们可以把 logging 与函数调用作为一个整体封装起来复用，以此消除冗余：\n",
    "\n",
    "    def smart_work_bar(data):\n",
    "        logging.info('begin call: work_bar')\n",
    "        work_bar(data)\n",
    "        logging.info('call doen: work_bar')\n",
    "\n",
    "这样一来，我们只需调用 smart_work_bar 即可：\n",
    "\n",
    "    smart_work_bar(some_data)\n",
    "\n",
    "    # others...\n",
    "\n",
    "    smart_work_bar(another_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 通用闭包\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "函数封装方案看上去已经很科学了，但美中存在不足。举个例子，当 work_foo 调用前后也需要打日志时，我们还需要再实现一个新函数 smart_work_foo ，尽管它的功能与 smart_work_bar 并无二致！\n",
    "\n",
    "我们观察发现， smart_work_bar 与 smart_work_foo 这两个函数的不同之处只是内部调用的函数不同。利用先前学到的 闭包 知识，我们实现 log_call 函数，将被调用函数参数化：\n",
    "\n",
    "    def log_call(func):\n",
    "        def proxy(*args, **kwargs):\n",
    "            logging.info('begin call: {name}'.format(name=func.__name__))\n",
    "            result = func(*args, **kwargs)\n",
    "            logging.info('call done: {name}'.format(name=func.__name__))\n",
    "            return result\n",
    "        return proxy\n",
    "        \n",
    "log_call 函数接收一个函数对象 func 作为参数，这是被代理函数；然后返回一个代理函数，即 proxy 函数。当代理函数 proxy 被调用时，先输出日志，然后调用被代理函数 func ，调用完毕后再输出日志，最后返回调用结果。 这样一来，不就达到通用化的目的了吗？——对于任意被代理函数 func ， log_call 均可轻松应对。\n",
    "\n",
    "    smart_work_bar = log_call(work_bar)\n",
    "    smart_work_foo = log_call(work_foo)\n",
    "\n",
    "    smart_work_bar(some_data)\n",
    "    smart_work_foo(some_data)\n",
    "\n",
    "    # others...\n",
    "\n",
    "    smart_work_bar(another_data)\n",
    "    smart_work_foo(another_data)\n",
    "    \n",
    "第 1 行中， log_call 接收参数 work_bar ，返回一个代理函数 proxy ，并赋给 smart_work_bar 。 第 4 行中，调用 smart_work_bar ，也就是代理函数 proxy ；代理函数先输出日志，然后调用 func 也就是 work_bar ；最后再输出日志。 注意到，代理函数中， func 与传进去的 work_bar 对象紧紧关联在一起了，这就是 **闭包** 。\n",
    "\n",
    "顺便提一下，可以覆盖被代理函数名，毕竟以 smart_ 为前缀取新名字还是显得有些累赘：\n",
    "\n",
    "    work_bar = log_call(work_bar)\n",
    "    work_foo = log_call(work_foo)\n",
    "\n",
    "    work_bar(some_data)\n",
    "    work_foo(some_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 语法糖"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "先回顾下面这段代码：\n",
    "\n",
    "    def work_bar(data):\n",
    "        pass\n",
    "    work_bar = log_call(work_bar)\n",
    "\n",
    "\n",
    "    def work_foo(data):\n",
    "        pass\n",
    "    work_foo = log_call(work_foo)\n",
    "    \n",
    "虽然代码没有什么冗余了，但是看是去还是不够直观。这时候，语法糖来了！\n",
    "\n",
    "    @log_call\n",
    "    def work_bar(data):\n",
    "        pass\n",
    "        \n",
    "请注意，开始 划重点啦 ！你可以这样认为： @log_call 这行代码的作用只是告诉 Python 编译器，在函数定义后面插入代码 work_bar = log_call(work_bar) ，仅此而已。不信？我们拿字节码说话！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 字节码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "对上面这个装饰器版的 work_bar 函数代码进行编译，我们可以得到这样的字节码：\n",
    "\n",
    "      2           0 LOAD_NAME                0 (log_call)\n",
    "                  2 LOAD_CONST               0 (<code object work_bar at 0x100cf2d20, file \"<dis>\", line 2>)\n",
    "                  4 LOAD_CONST               1 ('work_bar')\n",
    "                  6 MAKE_FUNCTION            0\n",
    "                  8 CALL_FUNCTION            1\n",
    "                 10 STORE_NAME               1 (work_bar)\n",
    "                 12 LOAD_CONST               2 (None)\n",
    "                 14 RETURN_VALUE\n",
    "                 \n",
    " 经过前面章节的学习，相信你对这些字节码已经非常熟悉了。接着，我们将虚拟机执行字节码的过程推演一番：\n",
    " \n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_24.png\" width=800 height=600> \n",
    "\n",
    "1. 第一条字节码将 log_call 函数加载进当前执行栈栈顶；\n",
    "2. 第二、三条字节码将 work_bar 代码对象和 work_bar 函数名加载到栈顶，为创建 work_bar 函数做好准备；\n",
    "3. 第四条字节码完成 work_bar 函数创建，该字节码执行完毕后，work_bar 函数便位于栈顶；\n",
    "4. 第五条字节码则以 work_bar 为参数调用 log_call 函数，并将 log_call 返回的 proxy 函数保存于栈顶；\n",
    "5. 接下来的 STORE_NAME 从栈顶取出 proxy 函数并保存到当前局部名字空间，它一般也是模块的属性空间；\n",
    "\n",
    "这下明白了吧？以后面试官再聊装饰器时，总算不会心虚了！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 求值装饰器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "先来猜猜装饰器 eval_now 有什么作用？\n",
    "\n",
    "    def eval_now(func):\n",
    "        return func()\n",
    "        \n",
    "看上去好奇怪哦，装饰器内部没有定义代理函数，算装饰器吗？\n",
    "\n",
    "    >>> @eval_now\n",
    "    ... def foo():\n",
    "    ...     return 1\n",
    "    ...\n",
    "    >>> print(foo)\n",
    "    1\n",
    "    \n",
    "这段代码输出 1 ，也就是对函数进行调用求值。 那么，这种写法到底有什么用呢？直接写 foo = 1 不行么？ 在这个简单的例子，这样写当然也是可以的。如果是更复杂的场景，就另当别论了。例如，初始化一个日志对象：\n",
    "\n",
    "    # some other code before...\n",
    "\n",
    "    # log format\n",
    "    formatter = logging.Formatter(\n",
    "        '[%(asctime)s] %(process)5d %(levelname) 8s - %(message)s',\n",
    "        '%Y-%m-%d %H:%M:%S',\n",
    "    )\n",
    "\n",
    "    # stdout handler\n",
    "    stdout_handler = logging.StreamHandler(sys.stdout)\n",
    "    stdout_handler.setFormatter(formatter)\n",
    "    stdout_handler.setLevel(logging.DEBUG)\n",
    "\n",
    "    # stderr handler\n",
    "    stderr_handler = logging.StreamHandler(sys.stderr)\n",
    "    stderr_handler.setFormatter(formatter)\n",
    "    stderr_handler.setLevel(logging.ERROR)\n",
    "\n",
    "    # logger object\n",
    "    logger = logging.Logger(__name__)\n",
    "    logger.setLevel(logging.DEBUG)\n",
    "    logger.addHandler(stdout_handler)\n",
    "    logger.addHandler(stderr_handler)\n",
    "\n",
    "    # again some other code after...\n",
    "    \n",
    "借助 eval_now 装饰器，我们的 logger 对象还可以这样来写：\n",
    "\n",
    "    # some other code before...\n",
    "\n",
    "    @eval_now\n",
    "    def logger():\n",
    "        # log format\n",
    "        formatter = logging.Formatter(\n",
    "            '[%(asctime)s] %(process)5d %(levelname) 8s - %(message)s',\n",
    "            '%Y-%m-%d %H:%M:%S',\n",
    "        )\n",
    "\n",
    "        # stdout handler\n",
    "        stdout_handler = logging.StreamHandler(sys.stdout)\n",
    "        stdout_handler.setFormatter(formatter)\n",
    "        stdout_handler.setLevel(logging.DEBUG)\n",
    "\n",
    "        # stderr handler\n",
    "        stderr_handler = logging.StreamHandler(sys.stderr)\n",
    "        stderr_handler.setFormatter(formatter)\n",
    "        stderr_handler.setLevel(logging.ERROR)\n",
    "\n",
    "        # logger object\n",
    "        logger = logging.Logger(__name__)\n",
    "        logger.setLevel(logging.DEBUG)\n",
    "        logger.addHandler(stdout_handler)\n",
    "        logger.addHandler(stderr_handler)\n",
    "\n",
    "        return logger\n",
    "\n",
    "    # again some other code after...\n",
    "    \n",
    "两段代码要达到的目的是一样的，但是后者显然更清晰，颇有独立 **代码块** 的风范。 更重要的是，日志对象在函数局部名字空间完成初始化，避免临时变量(如 formatter 等)污染外部的名字空间，特别是全局名字空间。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 带参数装饰器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "定义一个装饰器，用于记录慢函数调用：\n",
    "\n",
    "    def real_log_slow_call(func):\n",
    "        def proxy(*args, **kwargs):\n",
    "            start_ts = time.time()\n",
    "            result = func(*args, **kwargs)\n",
    "            end_ts = time.time()\n",
    "\n",
    "            seconds = end_ts - start_ts\n",
    "            if seconds > 1:\n",
    "            logging.warn('slow call: {name} in {seconds}s'.format(\n",
    "                name=func.func_name,\n",
    "                seconds=seconds,\n",
    "            ))\n",
    "\n",
    "            return result\n",
    "\n",
    "        return proxy\n",
    "        \n",
    "第 3 、 5 行分别在函数调用前后采样当前时间，第 7 行计算调用耗时，耗时大于一秒输出一条警告日志。\n",
    "\n",
    "    @real_log_slow_call\n",
    "    def sleep_seconds(seconds):\n",
    "        time.sleep(seconds)\n",
    "\n",
    "    sleep_seconds(0.1)  # 没有日志输出\n",
    "\n",
    "    sleep_seconds(2)    # 输出警告日志\n",
    "    \n",
    "然而，阈值设置总是要视情况决定，不同的函数可能会设置不同的值。如果阈值有办法参数化就好了：\n",
    "\n",
    "    def log_slow_call(func, threshold=1):\n",
    "        def proxy(*args, **kwargs):\n",
    "            start_ts = time.time()\n",
    "            result = func(*args, **kwargs)\n",
    "            end_ts = time.time()\n",
    "\n",
    "            seconds = end_ts - start_ts\n",
    "            if seconds > threshold:\n",
    "                logging.warn('slow call: {name} in {seconds}s'.format(\n",
    "                    name=func.func_name,\n",
    "                    seconds=seconds,\n",
    "                ))\n",
    "\n",
    "            return result\n",
    "\n",
    "        return proxy\n",
    "\n",
    "然而， @xxxx 语法糖总是以被装饰函数为参数调用装饰器。换句话讲，我们没机会来传递 threshold 参数。 这可怎么办呢？——只能再用一个闭包来封装 threshold 参数：\n",
    "\n",
    "    def log_slow_call(threshold=1):\n",
    "        def decorator(func):\n",
    "            def proxy(*args, **kwargs):\n",
    "                start_ts = time.time()\n",
    "                result = func(*args, **kwargs)\n",
    "                end_ts = time.time()\n",
    "\n",
    "                seconds = end_ts - start_ts\n",
    "                if seconds > threshold:\n",
    "                logging.warn('slow call: {name} in {seconds}s'.format(\n",
    "                    name=func.func_name,\n",
    "                    seconds=seconds,\n",
    "                ))\n",
    "\n",
    "                return result\n",
    "\n",
    "            return proxy\n",
    "\n",
    "        return decorator\n",
    "\n",
    "\n",
    "    @log_slow_call(threshold=0.5)\n",
    "    def sleep_seconds(seconds):\n",
    "        time.sleep(seconds)\n",
    "        \n",
    "这样一来，调用 log_slow_call(threshold=0.5) 将返回 decorator 函数，该函数拥有闭包变量 threshold ，值为 0.5 。 decorator 再装饰 sleep_seconds 。\n",
    "\n",
    "如果采用默认阈值，函数调用还是不能省略：\n",
    "\n",
    "    @log_slow_call()\n",
    "    def sleep_seconds(seconds):\n",
    "        time.sleep(seconds)\n",
    "        \n",
    "处女座可能会对第一行这对括号感到不爽，那么你还可以这样来改进：\n",
    "\n",
    "    def log_slow_call(func=None, threshold=1):\n",
    "        def decorator(func):\n",
    "            def proxy(*args, **kwargs):\n",
    "                start_ts = time.time()\n",
    "                result = func(*args, **kwargs)\n",
    "                end_ts = time.time()\n",
    "\n",
    "                seconds = end_ts - start_ts\n",
    "                if seconds > threshold:\n",
    "                logging.warn('slow call: {name} in {seconds}s'.format(\n",
    "                    name=func.func_name,\n",
    "                    seconds=seconds,\n",
    "                ))\n",
    "\n",
    "                return result\n",
    "\n",
    "            return proxy\n",
    "\n",
    "        if func is None:\n",
    "            return decorator\n",
    "        else:\n",
    "            return decorator(func)\n",
    "            \n",
    "这种写法兼容两种不同的用法，用法 A 默认阈值(无调用)；用法 B 自定义阈值(有调用)。\n",
    "\n",
    "    # Case A\n",
    "    @log_slow_call\n",
    "    def sleep_seconds(seconds):\n",
    "        time.sleep(seconds)\n",
    "\n",
    "\n",
    "    # Case B\n",
    "    @log_slow_call(threshold=0.5)\n",
    "    def sleep_seconds(seconds):\n",
    "        time.sleep(seconds)\n",
    "        \n",
    "用法 A 中，发生的事情是 log_slow_call(sleep_seconds) ，也就是 func 参数是非空的，这时直接调 decorator 函数进行包装并返回，因此阈值是默认的。\n",
    "\n",
    "用法 B 中，首先执行的是 log_slow_call(threshold=0.5) ， func 参数为空，直接返回新的装饰器 decorator ，关联闭包变量 threshold ，值为 0.5 ； 然后， decorator 再装饰函数 sleep_seconds ，即执行 decorator(sleep_seconds) 。 注意到，此时 threshold 关联的值是 0.5 ，由此实现了阈值定制化。\n",
    "\n",
    "你可能注意到了，这里最好使用关键字参数这种调用方式——使用位置参数会很丑陋：\n",
    "\n",
    "    # Case B-\n",
    "    @log_slow_call(None, 0.5)\n",
    "    def sleep_seconds(seconds):\n",
    "        time.sleep(seconds)\n",
    "        \n",
    "当然了， **函数调用尽量使用关键字参数** 是一种极佳实践，含义清晰，在参数很多的情况下更应如此。\n",
    "\n",
    "后来我又悟出另一种更加精炼的写法，亲你能否猜出？哈哈，我先卖个关子，答案在《面试题精讲》一节揭晓。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 智能装饰器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "上节介绍的带参数装饰器写法，嵌套层次较多。如果每个类似的装饰器都用这种方法实现，还是比较费劲的，也比较容易出错，脑子可能不够用呢。那么，有没有办法进一步进行优化呢？\n",
    "\n",
    "假设，我们有一个智能装饰器 smart_decorator ，用来修饰装饰器 log_slow_call ，便使其具备参数化的能力。 这样， log_slow_call 定义将变得更清晰，实现起来也更省事了：\n",
    "\n",
    "    @smart_decorator\n",
    "    def log_slow_call(func, threshold=1):\n",
    "        def proxy(*args, **kwargs):\n",
    "            start_ts = time.time()\n",
    "            result = func(*args, **kwargs)\n",
    "            end_ts = time.time()\n",
    "\n",
    "            seconds = end_ts - start_ts\n",
    "            if seconds > threshold:\n",
    "            logging.warn('slow call: {name} in {seconds}s'.format(\n",
    "                name=func.func_name,\n",
    "                seconds=seconds,\n",
    "            ))\n",
    "\n",
    "            return result\n",
    "\n",
    "        return proxy\n",
    "\n",
    "脑洞开完， smart_decorator 如何实现呢？掌握闭包和装饰器的执行机制，这其实也简单：\n",
    "\n",
    "    def smart_decorator(decorator):\n",
    "    \n",
    "        def decorator_proxy(func=None, **kwargs):\n",
    "            if func is not None:\n",
    "                return decorator(func=func, **kwargs)\n",
    "\n",
    "            def decorator_proxy(func):\n",
    "                return decorator(func=func, **kwargs)\n",
    "\n",
    "            return decorator_proxy\n",
    "\n",
    "        return decorator_proxy\n",
    "        \n",
    "smart_decorator 实现了以后，我们的设想就成立了！ 这时， log_slow_call ，就是 decorator_proxy (外层)， 关联的闭包变量 decorator 是本小节最开始定义的 log_slow_call (为了避免歧义，我们将其称为 real_log_slow_call )。由此一来， log_slow_call 支持以下各种用法：\n",
    "\n",
    "    # Case A\n",
    "    @log_slow_call\n",
    "    def sleep_seconds(seconds):\n",
    "        time.sleep(seconds)\n",
    "        \n",
    "用法 A 中，执行的是 decorator_proxy(sleep_seconds) (外层)， func 非空， kwargs 为空； 直接执行 *decorator(func=func, *kwargs) ，即 real_log_slow_call(sleep_seconds) ，结果是关联默认参数的 proxy 。\n",
    "\n",
    "    # Case B\n",
    "    # Same to Case A\n",
    "    @log_slow_call()\n",
    "    def sleep_seconds(seconds):\n",
    "        time.sleep(seconds)\n",
    "        \n",
    "用法 B 中，先执行 decorator_proxy() ， func 及 kwargs 均为空，返回 decorator_proxy 对象(内层)； 再执行 decorator_proxy(sleep_seconds) (内层)；最后执行 decorator(func, **kwargs) ， 等价于 real_log_slow_call(sleep_seconds) ，效果与用法 A 一致。  \n",
    "\n",
    "    # Case C\n",
    "    @log_slow_call(threshold=0.5)\n",
    "    def sleep_seconds(seconds):\n",
    "        time.sleep(seconds)\n",
    "\n",
    "用法 C 中，先执行 decorator_proxy(threshold=0.5) ， func 为空但 kwargs 非空，返回 decorator_proxy 对象(内层)； 再执行 decorator_proxy(sleep_seconds) (内层)；最后执行 decorator(sleep_seconds, **kwargs) ， 等价于 real_log_slow_call(sleep_seconds, threshold=0.5) ，阈值实现自定义！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 函数相关面试题精讲"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "函数机制是 Python 虚拟机的核心组成部分， **作用域** 、**名字空间** 、**闭包** 、**装饰器** 等基本概念均与函数机制紧密联系。\n",
    "\n",
    "正因如此，函数机制也成为 Python 工程师技术面重点讨论的内容。面试不仅考察后续人对相关知识点的理解，而且可通过类似 **装饰器设计** 这样的编码环节，考察候选人的程序设计能力。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. **Python 中有几个名字空间，分别是什么？Python 变量以什么顺序进行查找？**\n",
    "\n",
    "Python 总共有 4 个名字空间：\n",
    "\n",
    "- 局部名字空间 ( locals )\n",
    "- 闭包名字空间 ( enclosures )\n",
    "- 全局名字空间 ( globals )\n",
    "- 内建名字空间 ( builtin )\n",
    "\n",
    "Python 查找变量时，依次检查 **局部** 、 **闭包** 、 **全局** 、 **内建** 这几个名字空间，直到变量被找到为止。更多详情，请参考虚拟机部分 作用域 与 名字空间 以及函数机制部分 函数调用 相关章节。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. **如何在一个函数内部修改全局变量？**\n",
    "\n",
    "在函数内部用 global 关键字将变量声明为全局，然后再进行修改：\n",
    "\n",
    "    >>> a = 1\n",
    "    >>> def func():\n",
    "    ...     global a\n",
    "    ...     a = 2\n",
    "    ...\n",
    "    >>> print(a)\n",
    "    1\n",
    "    >>> func()\n",
    "    >>> print(a)\n",
    "    2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. **不使用 def 关键字的话，还有什么办法可以创建函数对象？**\n",
    "\n",
    "根据 Python 对象模型，实例对象可以通过调用类型对象来创建。而函数类型对象，理论上可以通过函数对象找到：\n",
    "\n",
    "    >>> def a():\n",
    "    ...     pass\n",
    "    ...\n",
    "    >>> function = a.__class__\n",
    "    >>> function\n",
    "    <class 'function'>\n",
    "    \n",
    "实际上，Python 将函数类型对象暴露在 types 模块中，可通过模块属性 FunctionType 访问到：\n",
    "\n",
    "    >>> from types import FunctionType\n",
    "    >>> FunctionType is function\n",
    "    True\n",
    "    \n",
    "函数对象包含以下要素，只要将这些要素备齐，即可调用函数类型对象创建新函数：\n",
    "\n",
    "- code ，代码对象；\n",
    "- globals ，全局名字空间；\n",
    "- name ，函数名；\n",
    "- argdefs ，默认参数值；\n",
    "- closure ，闭包变量；\n",
    "\n",
    "我们定义一个函数用于试验，它设置全局变量 value ：\n",
    "\n",
    "    >>> def set_global_value(v):\n",
    "    ...     global value\n",
    "    ...     value = v\n",
    "    ...\n",
    "    \n",
    "一开始全局名字空间并没有变量 value ，经过 set_global_value 函数设置后便可找到：\n",
    "\n",
    "    >>> value\n",
    "    Traceback (most recent call last):\n",
    "      File \"<stdin>\", line 1, in <module>\n",
    "    NameError: name 'value' is not defined\n",
    "    >>> set_global_value(10)\n",
    "    >>> value\n",
    "    10\n",
    "    >>> set_global_value(20)\n",
    "    >>> value\n",
    "    20\n",
    "    \n",
    "接下来，我们借助函数类型对象，创建一个与 set_global_value 类似的新函数 sgv ，只不过全局名字空间与我们指定的字典对象 g 绑定：\n",
    "\n",
    "    >>> g = {}\n",
    "    >>> sgv = FunctionType(set_global_value.__code__, g, 'sgv')\n",
    "\n",
    "这样一来，调用 sgv 函数设置的全局变量，将保存在指定的全局名字空间，即字典 g 中：\n",
    "\n",
    "    >>> g\n",
    "    {}\n",
    "    >>> sgv(10)\n",
    "    >>> g\n",
    "    {'value': 10}\n",
    "    >>> sgv(20)\n",
    "    >>> g\n",
    "    {'value': 20}\n",
    "    \n",
    "这个例子纯粹为了演示如何通过函数类型对象创建新函数，虽不具备实际项目用途，但对理解函数运行机制很有帮助。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. **4. 请介绍装饰器的运行原理，并说说你对 @xxxx 这种写法的理解？**\n",
    "\n",
    "装饰器用于包装函数对象，以达到修改函数行为的目的。\n",
    "\n",
    "装饰器是可调用对象，一般是函数，它接收被装饰函数 wrapped 作为参数，并返回包装后的函数。开发装饰器时，一般在装饰器内部定义一个嵌套函数 proxy 作为代理，proxy 通过闭包变量可以引用被包装函数 wrapped 。\n",
    "\n",
    "    def decorator(wrapped):\n",
    "        def proxy(*args, **kargs):\n",
    "            # 在这里调用被装饰函数wrapped\n",
    "            # 调用前后可以做一些额外的事情，\n",
    "            # 达到修改 wrapped 函数行为的目的\n",
    "\n",
    "        return proxy\n",
    "        \n",
    "嵌套函数 proxy 是实现装饰器的关键，它调用 wrapped 函数完成工作，调用前后可以做一些影响 wrapped 行为的事情。最后，proxy 函数作为包装结果返回给调用者。\n",
    "\n",
    "而 @xxxx 只是一个简单的语法糖，表示用 xxxx 装饰器修改被包装函数，与直接调用 xxxx 没什么区别。\n",
    "\n",
    "    @xxxx\n",
    "    def foo():\n",
    "        pass\n",
    "    def foo():\n",
    "        pass\n",
    "\n",
    "    foo = xxxx(foo)\n",
    "    \n",
    "因此，上面这两种不同的写法是等价的。在字节码层面，我们可以证实这一点：\n",
    "\n",
    "    >>> import dis\n",
    "    >>> text = '''\n",
    "    ... @xxxx\n",
    "    ... def foo():\n",
    "    ...     pass\n",
    "    ... '''\n",
    "    >>> dis.dis(compile(text, '', 'exec'))\n",
    "      2           0 LOAD_NAME                0 (xxxx)\n",
    "                  2 LOAD_CONST               0 (<code object foo at 0x101037ed0, file \"\", line 2>)\n",
    "                  4 LOAD_CONST               1 ('foo')\n",
    "                  6 MAKE_FUNCTION            0\n",
    "                  8 CALL_FUNCTION            1\n",
    "                 10 STORE_NAME               1 (foo)\n",
    "                 12 LOAD_CONST               2 (None)\n",
    "                 14 RETURN_VALUE\n",
    "\n",
    "    Disassembly of <code object foo at 0x101037ed0, file \"\", line 2>:\n",
    "      4           0 LOAD_CONST               0 (None)\n",
    "                  2 RETURN_VALUE\n",
    "                  \n",
    "偏移量为 6 的字节码 MAKE_FUNCTION 完成 foo 函数创建，而偏移量为 8 的字节码 CALL_FUNCTION 则调用了 xxxx 装饰器函数对 foo 函数进行装饰。\n",
    "\n",
    "虽然 @xxxx 语法与直接调用装饰器实际效果是一样的，但代码更为简洁，逻辑也更为清晰。\n",
    "\n",
    "广义的装饰器不仅可以用来修改函数，还可以用来修改类 —— 接收被装饰的类作为参数，返回包装后的类作为结果。另外，装饰器也不一定要对函数进行修改。例如 register 装饰器，它将函数注册到某个地方，然后原样返回：\n",
    "\n",
    "    def register(func):\n",
    "        # 将func注册到某个地方\n",
    "        # ...\n",
    "\n",
    "        return func\n",
    "        \n",
    "不少 Web 框架正是利用这个特性，优雅地进行实现路由注册。 flask 就是其中一例：\n",
    "\n",
    "    @app.route('/')\n",
    "    def index():\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5. **请设计一个装饰器，在函数执行时间超过指定值时输出日志**\n",
    "\n",
    "    # Case A\n",
    "    # 函数执行超过5秒输出日志\n",
    "    @log_show_call(seconds=5)\n",
    "    def some_func():\n",
    "        # ...\n",
    "\n",
    "    # Case B\n",
    "    # 默认函数执行超过10秒输出日志\n",
    "    @log_slow_call\n",
    "    def some_func():\n",
    "        # ...\n",
    "        \n",
    "前面介绍的方案不是最优版本，我还卖了个关子，现在终于可以揭晓谜底了！答案如下：\n",
    "\n",
    "    def log_slow_call(wrapped=None, seconds=10):\n",
    "\n",
    "        if wrapped is None:\n",
    "            def another_decorator(wrapped):\n",
    "                return log_slow_call(wrapped, seconds)\n",
    "            return another_decorator\n",
    "\n",
    "        def proxy(*args, **kwargs):\n",
    "            start_time = time.time()\n",
    "            result = wrapped(*args, **kwargs)\n",
    "            expired = time.time() - start_time\n",
    "            if expired > seconds:\n",
    "                logging.warning('call {} expires {} seconds'.format(wrapped.__name__, expired))\n",
    "            return result\n",
    "\n",
    "        return proxy\n",
    "        \n",
    "先说场景 B ，Python 将调用 log_slow_call(some_func) ，因此 wrapped 参数是 some_func ，seconds 参数是默认值 10 。由于 wrapped 非 None ，log_slow_call 直接返回 proxy 代理函数作为包装结果。\n",
    "\n",
    "proxy 通过闭包与 wrapped 和 seconds 变量绑定，而 seconds 参数是默认值 10 ，因此被包装函数 some_func 执行超过 10 秒就输出日志。\n",
    "\n",
    "而在场景 A ，当我们调用 log_slow_call(seconds=5) 时，由于被装饰函数没有提供，因此它不是用来装饰函数。相反，它用于生成新装饰器 another_decorator ，新装饰器通过闭包与指定的参数 seconds=5 绑定。\n",
    "\n",
    "当我们用新装饰器 another_decorator 装饰函数 some_func 时，它再次调用 log_slow_call 生成 proxy 函数。注意到，它调用 log_slow_call 时， 将变量 wrapped=some_func 以及闭包变量 seconds=5 作为参数传进去。\n",
    "\n",
    "proxy 同样通过闭包与 wrapped 和 seconds 变量绑定，而 seconds 参数是 5 ，因此被包装函数 some_func 执行超过 5 秒就输出日志。\n",
    "\n",
    "除了这两种用法，log_slow_call 还支持另一种用法，留待亲自行推演：\n",
    "\n",
    "    @log_slow_call()\n",
    "    def some_func():\n",
    "        # ...\n",
    "        \n",
    "由此，我们得到一个比前一节更精炼的实现方案。由此可见，对优雅代码的追求是无止境的"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6. **Python 中的闭包变量可以被内部函数修改吗？**\n",
    "\n",
    "我们通过一个简单的例子，来考察闭包变量的行为\n",
    "\n",
    "    def counter():\n",
    "        value = 0\n",
    "\n",
    "        def get():\n",
    "            return value\n",
    "\n",
    "        def inc():\n",
    "            nonlocal value\n",
    "            value += 1\n",
    "\n",
    "        return get, inc\n",
    "couter 函数用闭包实现了自增计数器，返回两个 get 以及 inc 函数，分别用于获取数值以及自增：\n",
    "\n",
    "    >>> get, inc = counter()\n",
    "    >>> get()\n",
    "    0\n",
    "    >>> inc()\n",
    "    >>> get()\n",
    "    1\n",
    "    \n",
    "两个内部函数通过闭包访问定义在外层函数 counter 中的 value 变量：\n",
    "\n",
    "- get 函数体没有对 value 进行赋值，因此 value 不是局部变量，编译器将其与外层函数的 value 绑定；\n",
    "- inc 函数对 value 进行赋值，因此需要通过 nonlocal 将它申明为闭包变量，否则编译器将把它当做局部变量；\n",
    "\n",
    "因此，通过 nonlocal 关键字，Python 是可以修改闭包变量的。注意到，nonlocal 是 Python 3 才引入的关键字，Python 2 没提供改关键字，也就无法修改闭包变量了。\n",
    "\n",
    "至于某个变量属于哪个名字空间，由以下规则决定：\n",
    "\n",
    "- 如果变量被 global 或者 nonlocal 关键字申明，那么它就是 全局变量 或者 闭包变量 ；\n",
    "- 如果变量在当前名字空间中被赋值，它属于当前名字空间；\n",
    "- 如果变量在当前名字空间中未被赋值，它属于外层名字空间；"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7. **请描述执行以下程序将输出什么内容？并试着解释其中的原因。**\n",
    "\n",
    "    def add(n, l=[]):\n",
    "        l.append(n)\n",
    "        return l\n",
    "    \n",
    "    print(add(1))\n",
    "    print(add(2))\n",
    "    print(add(3))\n",
    "\n",
    "程序执行结果如下：\n",
    "\n",
    "    [1]\n",
    "    [1, 2]\n",
    "    [1, 2, 3]\n",
    "\n",
    "这有点令人丈二和尚摸不着头脑，明明默认参数是一个空列表，为什么第 2 、 3 次调用后，列表都比预期中多一些数值呢？这一切得从 Python 函数的运行机制说起 —— Python 函数默认参数是如何实现的？\n",
    "\n",
    "Python 函数在创建时便完成了默认参数的初始化，并保存在函数对象的 __defaults__ 字段中：\n",
    "\n",
    "    >>> add.__defaults__[0]\n",
    "    []\n",
    "    \n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_25.png\" width=800 height=600> \n",
    " \n",
    " 当我们调用 add(1) 时，Python 虚拟机创建一个 **栈帧** 对象 PyFrameObject ，用于保存函数执行过程中的上下文信息。栈顶对象保存函数局部变量以及一个运行栈，Python 虚拟机负责从函数对象中取出默认参数并设置相关局部变量：\n",
    " \n",
    "  <img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_26.png\" width=800 height=600> \n",
    "\n",
    "当 add(1) 执行完毕，作为函数默认参数的那个 list 对象，就包含了一个元素 1 ：\n",
    "\n",
    "  <img style=\"float: middle;\" src=\".\\imgs\\C_04\\C4_27.png\" width=800 height=600> \n",
    "\n",
    "当我们再次调用 add (2) 时，Python 虚拟机还是从函数对象中取出这个 list 对象作为 l 的默认参数。因此，第二个 print 语句输出 [1, 2] 也就不奇怪了。\n",
    "\n",
    "总结起来，默认参数在函数对象创建时便完成了初始化，并保存在函数对象中。当函数被调用时，Python 从函数对象中取出默认参数，而不是重新初始化。因此，无论 add 函数被调用多少遍，默认参数总是同一个 list 对象。\n",
    "\n",
    "这多少与我的直观感觉相悖，因此尽量不要用可变对象作为默认参数，以避免一些潜在的 BUG 。如果实在需要以可变对象作为默认参数，我们换一种更的严谨写法：\n",
    "\n",
    "    def add(n, l=None):\n",
    "        if l is None:\n",
    "            l = []\n",
    "        l.append(n)\n",
    "        return l\n",
    "当 add 函数被调用时，如果参数 l 未指定，Python 自动使用默认值 None 。函数内部对参数 l 进行判断，如果它的值是 None ，便将其设为一个新的空列表。这样，add 函数的行为就更符合我们的预期了。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": false,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "1107px",
    "left": "0px",
    "top": "92px",
    "width": "405.547px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
